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STEPP CheckUst 



Six Steps to Evaluating Your Program 

Step I: Choosing an Evaluation Question Date Completed: 

[Meeting 1: Week 1 of Evaluation Year) 
Questions to Be Answered 
Part 1: Review the model of your program. 

Part 2: Raise questions about your program. What do you want to know? 

Part 3: Discuss effort, effectiveness, and efficiency in relation to your questions. 

Part 4: Identify a purpose and audience for the evaluation. 

Part 5: Choose the question your evaluation will answer. 

Step 2: Designing an Evaluation Date Completed: 

(Meeting 2: Week 2 of Evaluation Year] 
Evaluation Design 

Part 1: Consider possible evaluation designs. 

Part 2: Choose the design that fits your question and situation. 

Part 3: Define your sample. 

Step 3: Designing Measurement Instruments Date Completed: 

[Meeting 3: Week 3 of Evaluation Year) 
Outcome Measurement 

Part 1: Measuring the results of your program. 

Part 2: Deciding on background data. 

[Meeting 4: Week 4 of Evaluation Year] 
Implementation Measurement 
Part 3: Consider instruments for measuring implementation. 

Step 4: Building a Data Collection Plan Date Completed: 

[Meeting 5: Week 5 of Evaluation Year] 
Organizing and Collecting Data 

Part 1: Plan how to collect the data. How often will you collect it? Who will do it? 

Part 2: Pilot test instruments and collection plan. 

[Meeting 6: Week 6 of Evaluation Year) 
Troubleshooting Measures and Collection Procedures 
Part 3: Monitor field reports on measures and procedures. Adjust as needed. 

Step 5: Analyzing the Data Date Completed: 

[Meeting 7: Week 7 of Evaluation Year) 
Planning for Data Analysis 

Part 1: Choose analysis procedures. 

Part 2: Design tables and graphs for reporting results. 

Part 3: Begin collection of data. 

Part 4: Monitor data collection regularly. 

[Meeting 8: Week 45 of Evaluation Year) 
Analysis of Data 

Part 5: Analyze data. 

Step 6: Reporting the Findings Date Completed: 

[Meeting 9: Week 47 of Evaluation Year) 
Reporting the Findings 

Part I: Complete "dummy*" graphs and tables for reporting findings and plan presentation of data. 

Part 2: Role play presentation to target audience, adding data needed to respond to audience questions. 

Part 3: Report your findings to identified audiences. 

[Meeting 10: Week 50 of Evaluation Year] 
Making Use of What You've Learned 

Part 4: Adjust your program based on what youVe learned. 

Celebrate Your Accomplishments! 



Note: Meetings 1, 5, and 9 sliould involve your whole staff. 
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Introduction 



School administrators, teachers, parents and community leaders need to determine if their drug and 
alcohol prevention programs are making a difference. Questions need to be answered: Are attitudes 
about drug and alcohol use cha.^d? Is knowledge about harmful effects of drugs and alcohol in- 
creased? What factors are correlated to drug and alcohol use? Are drug and alcohol use reduced, 
prevented? 

These are some of the questions that can be addressed through a program evaluation. Answers will 
help to make programs accountable to their community and can provide managers important feed- 
back about their program operation. 

This handbook was developed for the purpose of providing program managers with a comprehen- 
sive yet easy-to-use tool to help their evaluation efforts. The handbook emphasizes program staff 
members working together as a team. It provides instruments and activities for determining program 
effectiveness, as well as documenting and monitoring the provision of services. 

We hope that readers of this handbook will view program evaluation as an important resource and 
also as a process with which they can feel comfortable and successful. 

Reed Bell, M.D. 
Director 

Office for Substance Abuse Prevention 
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Chapter 1 

Evaluation: Why Bother? 

Preventive Drug abuse preventive services seek to head off drug abuse problems before they occur, while treat- 

S^^'^'^^^s i^ent services are provided after an individual has developed a drug problem. Because they are provid- 

ed before a problem exists, preventive services seek to decrease the likelihood that individuals will 
abuse drugs by addressing and reducing factc^rs thought to increase the risks of abuse or by enhancing 
and promoting factors that are thought to inoculate people against drug abuse. In seekins to change 
the contributing conditions or risk factors for drug abuse, preventive services are provided in three 
ways: (1) direct services to target populations, (2) indirect services, such as sponsoring whole- 
community awareness programs, and (3) institutional change activities, such as campaigns to raise the 
drinking age, efforts to increase enforcement of drunk driving laws, or advocacy efforts to encourage 
schools to develop clear rules about drug use on campus. Preventive services often seek to alter the 
environmental conditions that contribute to substance abuse. 



The idea of preventive services is appealing. Given the high personal, social, and dollar costs of 
drug abuse, it makes sense to try to prevent drug problems rather than simply waiting for them to 
happen. Some people argue that preventive approaches should cost less than treatment services, which 
must remediate damages that have already occurred. Yet the evaluation of preventive services is not 
easy. To show that preventive services are effective, it is necessary to show that something did not 
happen (i.e., drug abuse) that would have happened had the services not been provided. 

When preventive services are not provided directly to individuals, evaluation becomes even more 
difficult. For example, the efforts of community-health projects seeking to prevent substance abuse 
through promotion of healthy habits cannot be measured by hours of service to individual clients, nor 
can their effects be determined by follov/ing up individual clients or checking their hospital and police 
records at periodic intervals after treatment. Another difficulty is that the timeframe for measuring ef- 
fects of preventive services may be lengthy. For example, the ultimate effectiveness of programs that 
help elementary school students develop skills to avoid trouble in hopes of preventing drug abuse may 
not be observable for several years. 

If the promise of preventive services is to be recognized and realized, however, agencies must plan, 
implement, and operate them to achieve the desired effects without undersired results. Without 
evidence of effects, government and private funding sources are often reticent to invest in preventive 
services; administrators have difficulty knowing how to divide resources among preventive and treat- 
ment programs; and staff have few guides for orienting the preventive services for maximum effec- 
tiveness. Thus, It is essential for the organization committed to performance to develop methods for 
monitoring and evaluating its own prevention activities. The purpose of this handbook is to provide 
you with the methodology to evaluate your prevention program. 



Think about "evaluation." What mental picture comes to your mind? When asked to make a men- 
tal picture of evaluation, people often conjure up amazing visions. Some of the visions of evaluation 
that have been offered by participants in evaluation workshops include: 

■ Flossing Your Teeth (You hatt to do it, but you know it should be done.) 

■ A Processed American Cheese Sandwich on White Bread vcith Mayonnaise and 
No Lettuce in an Airport Cafeteria (If you had your choice, you'd never look at 
it, let alone buy it.) 

■ Porcupines Making Love (There's a lot of promise, but it sure seems risky trying 
it.) 
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■ Doing Your Taxes When You Know You Wont Get a Refund (Something to put 
off until the last possible minute.) 

■ Doing a Stage Performance and Having Your Clothes Fall Off (Submitting to the 
risk of public exposure.) 

Close your eyes and picture your image of evaluation Now, take a minute to describe your im- 
age of evaluation in the space here. 



Why does evaluation, the process of seeing if we accomplished what we set out to achieve, generate 
such cynical thoughts in our minds? 

This may happen, in part, because the words "evaluation'' and ''accountability'' have gone hand in 
hand. In an era of tight resources for human service programs, getting and keeping the money needed 
to provide important preventive services to people has become a battle of epic proportions. In this 
context, evaluation can appear to those being evaluated as the ultimate strategic weapon in the hands 
of the enemy. Being evaluated can feel like giving an occupation force information on the 
whereabouts of your fellow fighters. French and colleagues (1983) report the case of a program direc- 
tor who was told by his State agency that "his program would be evaluated by his funding source, on 
its terms, with its evaluator, according to unofficial, legally questionable, and secret standards." 

Given these conditions, you're at risk if you do evaluation and in jeopardy if you don't. Par- 
ticipating in evaluation can take valuable time from service and administrative work; it can produce 
information that looks bad, information that may be put in the hands of those with the greaic; 
power to harm your program through budget cuts. On the other hand, if you resist evaluation, you 
open yourself up to charges of not caring about the quality of your services, or of failing to be fully 
accountable for the public funds you've been trusted to manage. 

What is a self-respecting human service organization to do about evaluation? One solution is to pull 
apart those synonyms — evaluation and accountability — and to think about and use evaluation dif- 
ferently. In the most blunt terms, it is possible to co-opt the concept of evaluation. A major dif- 
ference exists between being evaluated and doing an evaluation. 

It IS possible to use evaluation as a tool to answer those nagging que.'^tions you've had in the back 
of your mind about your own prevention program — Is this really working? Self evaluation allows 
you to ask the questions you want to ask to get answer^ you can use for your own purposes. This ap- 
proach allows you to drop the image of strategic information in enem> hands and to use that infor- 
mation yourself. 

What is this suggested approach to self-evaluation? First, it is important to think about what 
evaluation really means. Ev aluation is collecting and using informatiot to answer questions about 
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your program. It is a way of providing more information about the program than was available 
before. When you think about evaluation in this way, it becomes clear that the problems evaluations 
cause result not because people ask questions about their programs, but because of two corxerns* (1) 
who typically asks the questions and (2) why they want to know the information. 

When funding sources ask questions because they might want to cut off your funding, evaluation 
can be pretty threatening. When outside evaluators ask questions that provide information of 
marginal utility to the program, the process seems, at best, a waste of time. 

In self-evaluation, when you and your staff/team ask the questions, and the answers are used for 
your purposes to improve your prevention services, evaluation becomes a way to identify and solve 
problems, a way tc make prevention services more effective. 

The fact is that those of us engaged in drug abuse prevention constantly have questions about 
what and how we're doing. We may wonder: 



■ 



Is our school-br.3ed prevention curriculum really helping to prevent kids from us- 
ing alcohol and other drugs? 

■ Is our public service campaign reaching teenage girls at risk of smoking? 

■ Should we continue to offer awareness sessions for parents of high school students 
on recognizing signs and symptoms of drug use or provide skills for parents of 
fifth and sixth grade children to reduce the chances their children will experiment 
with drugs? 

■ How can we reach more parents with our parenting program? 

Prevention workers are continuously asking such questions. They must decide what prevention ac- 
tivities to initiate and maintain and how to organize themselves to provide them. Too often decisions 
must be made without relevant needed information. 

Evaluation, in this handbook, is r tool for informing your own decisions — a tool to use in prob- 
lem solving. Through the process outlined in this handbook, you will learn to do and use evaluation 
to identify and answer the questions you need to address in order to make better decisions in seeking 
to improve the effort, effectiveness, and efficiency of your preventive services. 

By gathering information for your own problem solving and troubleshooting, you will also take a 
proactive position on evaluation. Self-assessment and self-improvement activities become powerful, 
tangible evidence that you do care about the quality of your services. 

Such extra effort can also demonstrate program results and may ensure survival. Further, an effort 
to improve services based on identified problem areas demonstrates a commitment to accountability. 

Another reality of the social services marketplace is the competition to provide social services. 
Given expanding options for the investment of service dollars, how are you going to distinguish your 
services and your program from other programs? Schools are being pushed to offer drug abuse 
prevention programs, and parents are becoming an effective lobby for programs that cm demonstrate 
that they make a difference. What evidence can you produce to show that your program is superior 
to those offering alternative visions of prevention or intervention? Prevention program evaluation 
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enables you to present ''tested technology" and ultimately to speak with confidence about reductions 
in drug abuse associated with your program. 



Wojldn't it be nice to be able to make statements like this about your program? 

Overall, the smoking prevention program succeeded In reducing the Incidence of 
new smoking by about 58 percent. (Botvln and Eng 1982) 

Midway through seventh grade (only) 8 percent of the primary prevention and 
37.5 percent of control students had smoked In the previous mon.h. (Scklnke and 
Gilchrist 1983) 

The results . . . support earlier findings that programs addressing social Influences 
on smoking onset can effectively prevent Increased experimentation during 
preadolescent years .... The . . . smoking prevention program can be regarded as 
successful In getting young experimental smokers to quit earlier .... (Best et al. 
1984) 

These are actual statistics reported about prevention programs after self-evaluations were done. 
Statements like these encourage support for these programs. 

More important, a program of ongoing self-evaluation speaks to your ability to meet a community's 
changing need. By showing how its prevention activities address community substance abuse prob- 
lems, a dynamic, questioning prevention program can raise community concern and weave its pro- 
grams into the very fabric of school, family, and community life. 
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Chapter 2 



Don't We Need Experts? 

If these arguments for self-evaluation make sense, then one thing becomes clear, prevention pro- 
grams have to be responsible for their own evaluation. As a prevention program manager, you must 
ask the questions about your program if they're to be the right questions to infoim the decisions you 
ha ake in seeking to improve your drug abuse prevention efforts. Evaluation is, in a sense, 
you •■..carch and development arm. This handbook provides you one set of skills you need to be a 
successful manager of a prevention program. Clearly, evaluation isn't the only tool you need to be a 
good manager, but it is a powerful one! 

The STEPP (Staff/Team Evaluation of Prevention Programs) handbook is designed to be used by 
community groups and coalitions, schools, human service organizations, and any other groups work- 
ing to prevent drug abuse. You do not have to work in an agency or school to use this handbook. A 
group of concerned citizens can use this handbook as well as the staff of a human service agency. 
The handbook will walk you and your staff or team through six steps to a successful program evalua- 
tion. Chapters 3 through 9 will carry you fron^ the first questions you raise about your prevention 
program to a fully completed evaluation whose results are being used by your target audience to in- 
form decisionmaking. The STEPP handbook operates at both an individual and an organizational 
level. First, you will have an opportunity to learn the steps in the evaluation process by applying 
them to a question that you have about your piogram. You will also learn how to ipply the process 
in a staff/team evaluation involving your whole prevention group. 

If you are a program director or coordinator, you will be assisted in this staff/team process by 10 
meeting agendas that have been designed to maximize your staff or team's mvolvement n the evalua- 
tion process. 

Each chapter will be organized around a step in the evaluation process. Relevant meeting agendas 
will come at the end of each chapter. The handbook is designed to be interactive; application exer- 
cises provided in chapters 3-9, when completed, will document each of the steps. These exercises are 
boxed in to separate them from the narrative. In addition, the application exercises are combined in 
an evaluation think sheet (appendix A) that will help you in planning future evaluations. This hand- 
book is both a how-to-do-it text and the documentation of what youVe accomplished. If you take the 
time to complete each of the steps, you will, by the end of your first evaluation year, have instituted 
a process that will result in ongoing self-evaluation of the effectiveness of your prevention work. 

A time line is presented for a 52-week evaluation project. The actual length of your evaluation year 
may vary with the questions you select, but you will have the tools you need to complete the task. 

Starting anything new takes time; self-evaluation is no exception. 3y pacing yourself, stepping 
through the process of becoming a self-evaluating organization on a reasonable time line, you will be 
able to add self-evaluation to your group's agenda without overwhelming anyone. 

To show you how each step in the process can be applied to actual prevention programs, examples 
of the self-evaluation process as applied to prevention programs will be provided. 

Space limitations prevent answers to all the questions you may raise about program evaluation. 
Specifically, you may not find the exact measurement instrument, eval jation design, or statistical test 
you need in the text. When that happens, you will be referred to the Annotated Bibliography in the 
back of the handbook for recommendations of resources that provide more detailed and complete 
information. 
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"But dont we need experts? 



That is the first question heard at the suggestion of self-evaluation, and the answer is often "no!" 
The expertise you need to answer most questions about the effectiveness of your programs can be 
found in the pages of this handbook and in the easily accessible sources of evaluation design a.id 
statistical analysis information in the Annotated Bibliography. 

If you Jo have a question that seems beyond the scope of this handbook or need a design that is 
more complex than those discussed here, two additional resources should be pursued before yon con- 
sider hiring consultants. 

First, consider the skills and contacts of your own program team or staff. A major benefit of doing 
program staff/team evaluation is the sense of ownership among prevention workers that is created 
when you validate them by saying "We have a problem; how can we solve it together?" Staff or team 
input is invaluable in identifying the questions you need to ask and in ensuring that quality data will 
be collected to answer those questions. Further, you may find that people in your prevention group 
have skills that are not tapped by their current tasks. Members of your group may have a 
background in psychology, statistics, or research design that can be valuable in planning your evalua- 
tion. Or, you may discover that x team member has graphic arts skills that will be valuable in design- 
ing the format for reporting your findings, or a marketing background that will assist you in getting 
"the message" out. Your team or staff also represents an extended professional network. The hus- 
band, wife, or friend of a team member may have some special expertise that can greatly help your 
evaluation effort. A staff/team evaluation allows everyone in the prevention group to share the vision 
of self-improvement and a commitment to performance. 

Second, if you are located close to a college or a university that has a sociology, educational 
psychology, psychology, social work, biostati'stics, or public health department, don^t overlook the 
possibility of involving a professor or graduate student in your search foi knowledge about your pro- 
gram. University faculty actively search for interesting questions that can ue the source of research 
data for them or a research topic for one of their students. Your challenge may be their opportunity. 
But remember, this is your evaluation. You will want to maintain control of the evaluation and keep 
the focus on the questions you want to answer. 

To be successful in staff/team evaluation, you need three prerequisite qualities: curiosity, a desire 
to improve, and hard-headed caring. 

"How much will this cost?" 

Some groups avoid evaluation because they think they cannot afford it. It is true that rigorous 
evaluations of large-scale prevention projects can be expensive. However, it is also possible to do self- 
evaluation projects with a very small financial outlay, especially if you minimize your reliance on paid 
onsultants. Using this handbook, you will be able to formulate your own evaluation questions and 
develop your own evaluation designs and measurer :nt instruments. If you can enlist the assistance 
and support of your group and those organizations with whom you collaborate in prevention vou 
will be able to collect the data you need very inexpensively. Actual dollar costs can be limited the 
costs of typing and reproducing data collection instruments and entering and analyzing the informa- 
tion you collect. Important evaluation questions can be answered if you can identify the resources to 
cover these costs, which may total $1,000 or less depending on the scope of your project. 
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Before you begin, do you have questions that you expect this handbook to answer for you? List 
these questions here. 



1. 
2. 
3. 



You have just completed the first step in this handbook's self-evaluation plan. When you finish 
working your way through the STEPP handbook, return to this page to see if it has fulfilled your 
expectations. 



Chapter 3 



How Do We Begin? 

Step 1: Choosing an Eyaluation Question 

Part 1: The first step in successful evaluation is building ownership among all those who will be involved 

Renew the Mission in the process. If the service providers responsible for collecting data are not participants in the 

evaluation planning, they will find it difficult to find time to collect information that takes away from 
more immediate service responsibilities. Without reliable data, your evaluation planning is a waste of 
time If your board is not convinced that the question you seek to answer is an important one, they 
may fail tc enthusiastically back your efforts to spread your prevention work even if the results are 
positive*. Staff/team evaluation reflects a shared commitment to improvement of prevention and 
should involve the full range of participants in your prevention group or organization. 

Begin your STEPP process wit.i an all-team or all-staff meeting. It is here that you start to build 
the support necessary to successfully complete the evaluation.. Do not expect prevention workers, 
who may feel overburdened by case loads or program responsibilities, to respond to the word "evalua- 
tion'' with a standing ovation. A prevention manager must sell the merits of evaluation on the com- 
mon goal.^ of program quality and interest in improving prevention services. 

If the audience for this evaluation is broader than your own program staff, as it usually is, you will 
find it valuable to involve representatives of other organizations or agencies with an interest in your 
program to share in a meeting where you generate the questions you want your evaluation to answer. 
If the people you want to use your evaluation information share in helping to formulate the evalua- 
tion question from the beginning, you will build a positive reception for your findings. Strategies for 
expanding the audiences for your evaluation are discussed in the following excerpt from Hawkins and 
Sloma (1978). 
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Determining the Evaluation Questions 

(Adapted from Hawkins J David, and Sioma, Donald. Recognizing the organizational context: A strategy 
for evaluation research. Administration in Social Work 2(3), 1978.) 

Traditionally, program funders or top administrators have decided what questions they want answered m 
evaluation studies. Evaluators have responded with research designs to answer these questions By design- 
ing the evaluation to answer administrators' or funders' questions, evaluators have implicitly allied 
themselves in a social system with one set of participants, those who funded or commissioned the 
evaluation. 

This situation creates problems if funders,' administrators,' and social service practitioners' values goals 
a, . STn r- "°V'''.^""'=^'- cases, the latter group may perceive giving requested information 
as a threat to their activities. They may try to prevent funders or administrators from obtaining such infor- 
mation or subvert the evaluation in a number of ways including inaccurate information. 
We suggest that the process of defining evaluation questions be a negotiation process. 
As a first step in this negotiation process, all parties who have an interest in the program's operation - 
policymakers, funders, elected officials, program management, line staff, and even, in many programs, 
clients - should respond to two questions: (a) What is the program trying to accomplish? (b) What infor- 

e"fSely Sficiemiy? ^'""^ "'"^ '° '"""^ °' '° ^P''"'' P^"^^'"' "'"^^ 

On the basis of these questions, the evaluation manager should develop a tentative evaluation plan aimed 
at addressing the information needs oi as many of the concerned groups as possible. The tentative plan 
should contain several elements. First is a listing of those who provided input to the plan's content This 

isting gives members of the various audiences a preliminary sense of whether representatives of their in- 
TuflTJ^T . " '"f" consuMtd prior to development of the tentative plan. Second. 

he plan should contain a conceptual mof'.-l of the program or f.ystem of programs to be evaluated The 
Uiird portion of the phn should list the valuation questions raised by participants that the evaluation pro- 
pose to address and briefly describe the methods that will be used to obtain answers to these questions 
Unally, the evaluation plan should contain a listing of questions raised by potential audiences that the 
evaluation does not propose to answer using the proposed evaluation plan. This listing of questions not ad- 
dressed provides various actors with a clearer understanding of the limits of the proposed evaluation study. 
What questions should be included, and what evaluation questions should be omitted? The first rule of 
thumb would be to include questions that were asked by members of several potential audiences of the 
evaluation. To the extent that many consumers agree on the importance of finding the answer to a par- 
ticular question, the evaluation manager can increase the likelihood that the evaluation study will be seen 
as properly focused and useful. We also suggest that the evaluation manager be careful to include at least 
one quest.on asked by representatives of each of ihe audiences. In this way. the evaluation manager 
sabotagf it audience wii; see the study as totally without merit and attempt to 

The experience of the authors in designing evaluation projects using this process suggests that total 
disagreement betvyeen funders and program personnel about evaluation questions may be rare Where 
agreement is possible, several benefits result from proceeding as we have outlined. First, to the extent that 
coinmoTj questions for the evaluation are identified, the evaluation manager increases the likelihood of staff 
cooperation in the use of investigatory tools. Because the evaluation investigates staffs concerns, staff are 
more likely to agree to steps to aid the .tudy. Thus the process should ultimately allow evaluation 
managers to increase their methodological rigor. Second, the evaluation is cleared, at the outset, from the 
charge of irrelevance, of having focused on the wrong questions. By making input from all interested par- 
ies part of an explicit, documented negotiation process, the evaluator is able to place responsibility on 
those critical of the questions addressed for not taking a more active role in designing the study when it 
was offered. Third, the process has a clear developmental function. It encourages practitioners to define 
Clearly what their programs seek »o accomplish and what results they expect and identifies gaos between 
various parties conceptualizations of the program and program purposes 
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Most people who work to prevent alcohol and other drug abuse come to that work with a sense of 
mission, a vision of a better world. Your group has been drawn to your prevention program by a 
promise that their work under your guidance will help to create a community/society where fewer 
people are harmed by the abuse of drugs. However, sometimes something happens in the first few 
years of working on any tough problem that can wear the glitter off the goal. People begin to see 
themselves in the more confined perspective of "my job." People can lose sight of their ideals as they 
slug away at the v^ry real chores of linifig up refreshments for parenting classes, convincing teachers 
that they can "sacrifice" 30 minutes of their class time once a week for a drug education program, or 
arguing with ^he local TV slation manager to run prevention spots. It is important to regularly renew 
the common mission that brought you all together. An evaluation talks to the goals that are common 
among evaluation workers — the need to know that what prevention workers are doing has made a 
difference in the lives of people. 

An excellent vehicle for renewing the mission and putting your evaluation effort in the context of 
accomplishing that mission is the use of a simple visual model of your prevention program to be 
evaluated. This model shows your actual current program and what you expect it to achieve. The 
model need not be highly detailed. A useful format is a simple systems model (see figures 1 and 2). 

Figure 1. — Program Model 



The model should show resources, the program of services offered using these resources, immediate 
results you expect to follow your services, and desired outcomes over the longer term. Resources are 
the people, money, contacts, and connections that you put together into a program of services. The 
program of services consists of the actual prevention activities you offer. The immediate results your 
prevention program seeks to achieve may be better skills among teenagers in refusing drug offers or 
improved family communication. Ultimately, you expect these immediate results to reduce drug use or 
abuse among your target population. This reduction is the desired outcome of your prevention work. 

Figure 2 illustrates a program model for a parent-focused prevention program. 
Figure 2. — Program Model: Parent Prevention Program 

Resources Program of Services Immediate Results Outcomes 

Trainers Five weekly Parents will: Fewer children will 



Resources 



Program of 
Services 



Immediate 
Results 



Outcomes 



Curriculum package 



PTA sponsorship 
and volunteer 
recruiters 



parenting classes 



1. Establish clear 



family expectations 
about alcohol and 
other drugs in their 
families. 



experiment with 
alcohol, marijuana, 
or tobacco before 
leaving junior high. 



2. Teach their children 
how to avoid trou- 
ble and still have 



fun and friends. 



3. Keep family conflict 
from getting )ut of 
hand. 



4. Strengthen fai.iily 
bonding by increas- 
ing children's loles 
in the family. 
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Presenting a visual model of your program serves two important purposes. It focuses your group on 
the common goals of your program while showing members how their individual contributions fit ^n 
the bigger picture, and it provides a framework for raising specific questions about the program ^' 
What do we need to know about our program and how it is working?" For example, a question tliat 
might be raised about the parenting program in figure 2 is: "Did parents who participated in the pro- 
gram actually establish clear family expectations about alcohol and other drugs?" In other words did 
the program of parenting classes actually produce desired immediate results? 

At this point, draw a simple model of your prevention program following the example displayed in 
figure 2. In this model, show the following: 

■ The resources you use to provide prevention services 

■ The program of services 

■ The immediate results you expect your services to achieve 

■ The longer term outcomes you think will ultimately follow 



Part 2: 

Raise Questions 
about Your 
Program 



Resources 



A Model of Our Prevention Program 

Program of Services Immediate Results 



Outcomes 



Once everyone understands the model that shows your program you're ready to engage in the 
most important step of any evaluation ^ identifying the questions you want to answer. What do we 
need to know to improve our program, our service delivery, our outcomes, etc ... ? 

Think about the nagging questions you personally have about your program or agency'^ Write tv>'o 
or three of them down in the space here. 

I Wish I Knew 



1. 




2. 




3. 
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Part 3: 

Discuss Effort, 
Effectiveness, and 
Efficiency In 
Relation to 
Your Questions 



As you walk through the STEFP handbook to familiarize yourself with the steps in self-evaluation, 
you will probably seek to answer one of the questions you've come up with here. In using the 
staff/team evaluation process, however, you may want to avoid coming to a meeting for identifying 
questions armed with the evaluation question you want to address. You may find that a question that 
is important to you is viewed as less important to the rest of your group. Or, as other questions are 
brainstormed by your group, you may see a question that is more important than the one you initial- 
ly identified. 

The process of brainstorming questions, in itself, will identify a broad range of concerns. That's 
healthy! It testifies to a staff/team's desire for improvement. It is also grist for an important discussion 
of program priorities. What are the priority questions that we need to answer? Which questions seem 
to go together? What do we really need to know to make the important decisions we face as a group 
or organization? Through this sharing of concerns, your prevention group will walk in each other's 
shoes, you will strengthen the bond between individuals, you will validate the importance of all 
members of the group, and, most important, you will build group commitment to your program while 
you are planning for self-evaluation. 

Look at the questions you have jotted down about your program. Evaluation questions can be ask- 
ed about any of the parts of a prevention program. 

Figure 3. — Relationship of Effort, Effectiveness, and Efficiency to a Program Model 



Resources 



Program of 
Services 

I 1 



Effort 



Immediate 
Results 



Outcomes 



J 



Effectiveness 




•Efficiency 



Program 
Information 
Aspect 

Look at the program model in figure 3. One kind of question you may have is about program ser- 
vices: "What services did we actually provide and to whom?" Such questions are about program ef- 
fort. An example of an effort question is: "How many seventh grade students at Sylvan Junior High 
were trained to use refusal skills this year?" An effort question about a parent training program is: 
"On the average, how many of the five sessions did the parents attend?" 

Another question you hiay ask is whether you achieved the immediate results you wanted. As 
noted earlier, you may ask: "How many parents in our prevention class actually talked with their 
children about family rules on alcohol and drugs following the session?" This is a question of program 
effectiveness in achieving immediate results. In this example, the outcome the program sought was to 
increase family communication on expectations around drug use. 

You may also want to know if the desired outcomes of your prevention work were achieved. For 
example: "Were rates of drug initiation among junior high school students lower in families that 
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attended the prevention class than in families that did not?" This is also a question of program effec- 
tiveness, a question of effectiveness in changing drug use outcomes. 

A third kind of question is about rhe efficiency of your prevention work. This includes cost-per-unit 
of service and cost related to benefits achieved. For example, you may ask: "How much did it cost 
per student to provide our drug abuse curriculum in the schools?" This is a question of the relation- 
ship between resources and program services (cost-per-unit of service). What was the cost in resources 
to produce the services that resulted? Alternatively, you may want to know how much it cost to 
reduce the rate of smoking among sixth gr«d.' students by 25 percent. This u a question of the rela- 
tionship between resources, program service., and outcomes (cost-benefit analvsis). Often, before you 
can answer efficiency questions, you must have the answers to effort and effectiveness questions. 

Look at the questions you jotted down earlier: 

■ Do you see any effort questions? 

■ Do you see any effectiveness questions? 

■ Do you see any efficiency questions? 

Some of your questions may not fit. They don't seem to be questions about program effort, effec- 
tiveness or efficiency. Perhaps you asked "Where can we get the resources to hire moi3 parent 
trainers.' or How can we convince the legislature to fund our program for next year?" 

Not all your questions are program evaluation questions. Evaluation questions ask how well the 

r;°?T?DD "'^""''^ '^'^^'^ ""^ ""^^^'""^ that can be answered by following 

the S I EPF process. ^ 

In order to lead your staff or team in identifying evaluation questions for self-evaluatior, it will be 
useful to think of some effort, effectiveness, and efficiency questions of your own. 

Identifying Questions of Effort 

Measuring effort involves the documentation of staff time and resources invested in delivering ser- 
vices and counting the services delivevd. This measurement activity is frequently referred to as 
monitoring Monitoring can inch.Ht the recording of quantitative (How many clients did we serve'') as 
well as qualitative (How were services organized or provided?) information. 

Many organizations already have systems in place to monitor their prevention services If the infor- 
mation generated by a monitoring system is to be used for program improvement, it is important to 
find some incemive for people to record the information accurately. Ensur.ng the collection of the re- 
quired information by mandating it as an additional job requirement probablv will not guarantee good 
reporting and may not provide information in which you have confidence. The STEPP process of in- 
volving staff in planning for information collection, and thinking about how the information can be 
used to enhance services, should uicourage more accurate monitoring in your organization. Such a 
process will also show that the administration is truly interested in the information for improving the 
program rather than simply for monitoring staff compliance with their job descriptions. 

Effort evaluation is sometimes called process evaluation. It measures the process (program of services) 
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that leads to immediate results and long term outcomes. Refer back to your prevention program 
model, and take a few minutes to jot down two questions >ou have about yonr program of services 
I e.v your program's efforts 

Here are some examples to get you started. 

■ How many awareness sessions did we provide to community groups last year? 

■ How manv teachers did we train to use drug prevention materials in their classes? 

a How much of our preventive serv.ce time is spent in planning? How much time in 
service delivery? 

■ How many young people are we reaching through our peer-to peer program? 

■ What groups or organizations are we serving or collaborating with in our preven- 
tive services? 

■ Are we doing what we planned in the prevention area? 

B How manv families received prevention workbooks through our program last year? 
Effort Questions Abotit My Program 



1. 

1 



Identifying Questions about Effectiveness 

Perhaps the most important question about any prevention program is: "Did it work?" You may 
want to know. "Did our peer counseling program reduce alcohol use in the school? Did the teacher 
training project prevent drug experimentation among seventh grade students?" These are questions of 
program effectiveness. To answer these questions, which ask whether the program caused the desired 
outcomes, usually requires an experimental design by which it can be determined that the interven 
tions led to the observed results. 

A closely related question can be answered much more easily, however. This is the question of 
what happened to persons or conditions that were targeted by a program. For example, "Did the 
students of teachers wi ) used the prevention curriculum in their classrooms report lower levels of 
drug use?" An answer to this question does not require proof that the desired outcomes were caused 
by the program but simply asks if these outcomes were achieved. This type of information has been 
called outcome enumeration, which tells us whether things got worse or better after the program was 
initiated. 

Answering effectiveness questions is called an outcome evaluation. Outcome evaluations assess 
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either the immediate results or longer term outcomes of a program or both. 

A difficulty in assessing outcomes of prevention efforts is created by the fact that the desired 
results may not be observable for some time. As an example, consider a training program in refusal 
skills offered to third and fourth grade teachers as a drug abuse prevention strategy. Students of thc;>c 
teachers might not be expected to experiment with psychoactive drugs, even if not exposed to the 
program, for 4 years or more. Self-evaluating prevention organizations may not have the resources to 
follow these students over a long period to measure their future drug use. 

Two implications are important. First, even without assessing the ultimate outcome of student c'rug 
use. other mtermediate results can be assessed. Did teachers learn the techniques presented? Did 
students learn to use the skills taught? Were their attitudes toward drug use or their intentions 
regarding future drug use affected by the program? Even without long-term outcome data, answers to 
these questions can indicate how well the teacher training program is achieving immediate results and 
how it can be improved. Second, when long-term outcomes cannot be assessed by your organization, 
it becomes important to select prevention techniques that have either a strong conceptual foundation 
or have been shown by research to be promising or effective in achieving their goals. It is not suffi- 
cient for the group that seeks to launch an effective prevention program simply to offer those services 
with v/hich staff are familiar or comfortable, and to assume optimistically that they wili result in ef- 
fective prevention. Prevention services should address known risk factors for drug abuse and/or use 
approaches that have demonstrated positive results if t'le expected outcomes are unmeasurable or will 
occur only far in the future. (For summaries of known risk factors for drug abuse and their implica- 
tions for prevention, see Hawkins et al. 1985 and 1986.) 

The fact that prevention efforts may not directly serve individual clients also can make it difficult 
to assess the results of prevention activities. Indirect services, such as teacher training or advocacy ef- 
forts designed to develop alternative programs for students, may seek to prevent drug abuse. Since 
training and advocacy efforts do not directly serve individual students, however, it becomes more dif- 
ficult to use information about those served to discover program results. Yet measurement strategies 
(chapters 5 and 6) are available to provide effectiveness information in spite of these difficulties. 

Effort and effectiveness questions are closely linked. If you measure only the effort of your pro- 
gram, without assessing effectiveness, you will know how much work you have done. You will not 
know, however, if your work made any difference. 

If you measure only effectiveness, without measuring the effort that produced the observed results, 
you might not be abie to identify exactly what produced those results. 

Nevertheless, questions of effectiveness are fundamental que^^tions in and of themselves. By answer- 
ing effectiveness questions, you can discover whether risks for drug abuse and drug abuse itself are 
being reduced or prevented by your prevention efforts. Take a few moments to write down two ques- 
tions you have about your prevention program's effectiveness. Here are some examples to get you 
started. 

■ How effective is our holiday share-a-ride program in preventing driving while 
under the influence of alcohol or other drugs? 

■ Do more students than before perceive health risks associated with marijuana use 
following implementation of our new school-based prevention curriculum? 
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■ How effective is our parenting program in preventing early experimentation with 
drugs among children of participants? 

Effectiveness Questions About My Program 



1. 
2. 



Identifying Questions about Efficiency 

Efficiency is a question of the cost of reaching your desired outcomes. That cost can be in terms of 
dollars, staff time, curriculum materials, planning time, client investment, or any other measurable 
program resource. It is important to know that your diug education program costs a school district 
S157 per classroom to implement. It is equally important to know that a competing curriculum costs 
only $112 per classroom to implement. Which curriculum is more efficient will depend on the rela- 
tionship between effort (curriculum costs and teacher time) and effect (relative reduction in student 
drug use). 

Efficiency studies imply comparisons with standards. A prevention program can be compared with 
other programs (external! or previous efforts (internal), or comparisons can be made among ap- 
proaches. For example: Is it more efficient to have our prevention workers or classroom teachers pre- 
sent prevention curriculum in schools? While the costs of having prevention workers from your group 
teach the curriculum may be higher, they may also produce more positive effects than regular 
classroom teachers because of their commitment, knowledge, and skills in delivering tlie curriculum. 
This is a question of efficiency. 

Efficiency questions are of three types: Are resources used efficiently to maximize services? Are ser- 
vices efficient in producing immediate results? Does achieving these immediate results produce the 
desired long-term outcomes efficiently? 

Take a few moments to write down two efficiency questions you have about your prevention pro- 
gram. Here are some examples of efficiency questions to get you started. 

■ Does peer counseling or our refusal skills curriculum result in the larger 
reduction in students' intention to use drugs per-dollar-invested? 

■ Which parent recruitment strategy, student poster contests or a friendly letter- 
writing assignment, is the most efficient way to bring parents into our program? 

■ Given our limited funds, is it more efficient to serve every other student referred 
to our peer assistance program, or to serve half of them now and delay treatment 
to the other half? 
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Efficiency Questions about My Program 




Mastering the Three E's 

Here is a test of your ability to identify effort, effectiveness, and efficiency questions. 

Place an A by a sentence if it is asking an effort question. 
Place a B by a sentence if it is asking an effectiveness question. 
Place a C by a sentence if it is asking an efficiency question. 

1. Are parents satisfied with our elementary school drug education program? 



2- ^^^'^^ recruitment approach yields the most volunteer crisis line workers at the least 

cost? 

3. Does student participation in our peer counseling program reduce the reported use of 

marijuana and inhalants? 

4. How many parent classes did we provide in the 1985-86 school year? 

5. Which was the most cost-effective recruitment strategy for parent participation in 

classes, food raffles or free child care? 

6. Do police patrols reduce the number of alcohol-related accidents among young 

people? 

7. Did radio advertising or community Hyers do the best job of advertising our com- 
munity drug-free day? 

8. How much influence did classroom visits by professional athletes have on student at- 
titudes about drug abuse? 

Answers: B,C,B,A,C,B,C,B 



How did you do? If you answered seven or eight questions correctly, you're ready to lead your 
staff in a brainstorming session to identify your organization's questions for self-evaluation If you 
missed more than two answers, review the discussion of the "three E's" to reinforce your understand- 
ing of effort, effectiveness, and efficiency. 
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Which Questions Are Most Important? 



Part 4: 
Identifying a 
Purpose and 
Audience for 
the Evaluation 



By now youVe identified several questions about your program that you might answer through the 
STEPP process. Choosing which of these questions to actually address is the next self-evaluation task. 
You should follow a simple rule here. Answer only questions that will provide information that you 
will use. Before you set about designing a self-evaluation plan, you should know how you plan to use 
the information provided. 

With the exception of doctoral dissertations, little market exists for questions that have an uniden- 
tifiable utility and an audience of one. When you choose a question for your evaluation study, you 
want a question that has stature — a question that will repay the effort you and your team will in- 
vest in obtaining its answer. To make sure you have a question worth answering, compare the top 
priority questions you're considering in light of these three: 

1. What do we need to knov/ about our program to solve a problem we have 
identified? 

2. When we have the information, who will be interested in knowing it? 

3. Why do ' e need to know this? What can we do differently with this new 
information? 

The purpose/audience matrix in figure 4 will help you decide whether you have a use for the infor- 
mation you might gather in answering the questions youVe listed, and who your intended audience 
will be. If you have decided that funding is a major problem of your prevention work, but the ques- 
tion you've chosen would not be of interest to those who might provide such funding, you should 
consider a different question. If you are considering more than one question to be answered in this 
year's evaluation, complete the matrix for each question as a means of deciding which question will 
best serve the goals of your organization. 
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Figure 4.— Purpose/Audience Matrix 



Before deciding on an effectiveness question, identify why the information is needed. Why do you 
need to answer this question? What could you do with the information? This matrix attempts to help 
you clarify answers to th^^e questions bv outlining five possible uses to which information may be 
put in an agency and seven possible a\iiences to whom the information might be addressed. The list 
is not exhaustive. You ir* v,ant to add another purpose or audience that you think is important, 

Place an "x" in each box of the matrix that represents the intersection of a purpose and audience for 
which you believe the answer to your question will be useful. 



Management 
Control and 
Planning & Operational 
Development Decisions 



Purpose 



Secure 
Funding 



Public 
Relations 



099 



Management 



Service 
Providers 



Funders 



Other 
Agencies 



Public or 
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Participants 
or Clients 



999 



The purpose/audience matrix identifies what you will do with the information you have gathered at 
the er'^ of your evaluation. It allows you and your team to clarify the purpose of activities that may 
be tedious at times. If you don't know how you will use the information, don't bother to do a self- 
evaluation to answer the question! 

Spending time on generating and defining a question for self-evaluation is a worthwhile investment, 
because good questions are the key to good evaluations. If you know how you will use the informa- 
tion you get in answer to your question, then the STEPP process will be worth the investment. 

Part 5: Now that you know the kinds of questions you may want to ask and the uses to which you could 

Choose the Question put the answers, you're ready to plan for your first group meeting on self-evaluation. Prepare for the 
Your Evaluation meeting by completing a simple program model (see figure 1) for your own prevention program or 

Will Answer programs. Then review staff/team agenda 1 at the end of this chapter in preparation for the meeting. 

Be prepared to suggest the questions you've already come up with and to listen to the questions 
others suggest for self-evaluation as you finalize the one question you will choose for your self- 
evaluation project. 

If this is your first self-evaluation, we suggest that you and your staff/team start with an effec- 
tiveness question. By answering an effectiveness question, you will learn whether your program made 
a difference in reducing risk factors or in preventing drug abuse. By focusing on an important "bot- 
tom line" question, you will be able to maintain enthusiasm for the self-evaluation project. The re- 
mainder of this handbook is designed to help you plan and carry out a self-evaluation project to 
answer an effectiveness question about your prevention program. 

What is the effectiveness question you've chosen for your evaluation? 

Write *^*THE QUESTION*** here. 

************************** *********************************^ 
* 



****************************************************************************'********* 
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STAFF/TEAM AGENDA 1 
QUESTIONS TO BE ANSWERED 



ATTENDING: For a small program, the whole staff or prevention team. For a larger program, 
representatives from each unit. 

TIME: Schedule 2 hours. 

OBJECTIVE: To reach consensus on a major question you will answer about your program by in- 
volving total staff/team or unit representatives in listing and prioritizing information needs. 

I. Renew the mission. Each program should have a model — a schematic drawing of the goals of 
the program and the program linkages that enable those goals to be accomplished. This model helps 
each staff member to know how liis or her part in the process helps achieve the intended outcomes of 
the program. The model gives your program a clear vision of its mission. 

II. What do you need to know to improve your program, your service delivery, or your impact? 
(Brainstorm, using a chalk board, newsprint pad, or overhead projector.) 

III. Which of these questions can logically be combined? 

IV. Categorize the questions as questions of effort, effectiveness, or efficiency. 

V. Prioritize the questions of greatest interest. (You will not have the resources to answer all the 
questions this year. A prioritized list will allow you to answer the most important question first.) 

VI Share the process of staff/team evaluation with your staff. Review the six-step monitoring 
chart/time line. Are these questions important enough to your success that your group is ready to pay 
the time cost to answer one of them? 

If the answer is yes, continue to topic Vll. If the answer is no, determine whether the reluctance is 
an energy issue or a matter of having not identified a truly important question. If the question is not 
important, return to the brainstorming step and start again. If staff concerns center on the time that 
an evaluation will consume, investigate just one question this year. 

VII. Decide what youVe going to do with the information you will generate by the end of the year 
and who you want to share it with. This decision will allow you to design an evaluation that will pro- 
vide information that will be used. (Use the purpose/audience matrix in chapter 3 to help guide this 
discussion. 

VIII. Congratulate your staff or team: ^ small celebration is in order.) Set the date for your next 
meeting, and ask each member to come to the meeting with a list of resources both within the team 
and outside the group that might be used to help ^ is evaluation effort. 
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Chapter 4 



Evaluation Design 

Step 2: Designing an Evaluation 

An evaluation design allows you to answer the question "Compared to what?" To know if your 
program has made a difference, you must have some standard for comparison. 

Have you prevented drug abuse? The answer to this question depends on what you define as your 
standard of success. Would you say your school-based prevention program was a success if 5 percent 
cf the high school seniors in your community smoked marijuana daily? What if you knew that in 
1979, 11 percent of the high school seniors in the United States smoked marijuana daily (20 times a 
month or more). Then would you say your program was a success? What if you knew that by 1985 
only 4.9 percent of the seniors in the United States were daily marijuana users (20 times a month or 
more). Then would you view your program as a success? 

All these comparisons are possible; all the data are correct. In the absence of a standard for com- 
parison, how can you know if you're making progress in preventing drug abuse? You must compare 
your outcomes with something in order to decide if your prevention efforts have worked. 

The most obvious and appropriate standard for comparison is to compare rates of the drug abuse 
problem after your prevention efforts with what things would be like if your prevention program 
weren't there, i.e., to ask "What would things have been like without us?" 

For example, if you implement an alcohol abuse prevention curriculum in the eighth grade and 
find at the e' d of the year that the mean rate of alcohol use for students who experienced the cur- 
riculum was two times per week, is your program a success or a failure? It's hard to tell. Two times a 
week may seem to be a high rate. However, the school counselor may have placed all problem 
drinkers in your classes because she knew they would be receiving remediation for their problems. 
Perhaps the students v/ho reported drinking twice a week after your program were drinking every day 
at the start of the school year. Without a means of comparison such as a pretest (mean rate of drink- 
ing before the program), you would not know that alcohol use had been reduced by 60 percent after 
a year's exposure to your curriculum. Can you say, then, that your curriculum "caused" a 60-percent 
reduction in student drug use? Not until you have made other comparisons. It could be that many 
eighth grade students in this school were heavy drinkers at the start of the school year, but teenage 
alcohol-related traffic deaths during the winter holidays led to a 60-percent reduction in alcohol use 
throughout the student body. 

Evaluators call these historical effects— little events in time that change other things. Another 
example of a historical effect could be the institution of school locker checks or urinalysis screening 
procedures for suspected users in school, events that also may have reduced drug use in the school. 
There are possible explanations other than your prevention program for the outcomes you see, as 
discussed in the accompanying box. (These seven threats to validity were identified by Campbell and 
Stanley 1963.) 
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S^^ven Threats to Internal Validity 

HISTORY: History refers to the unplanned events thac occur between the time you take your 
first measurement (pretest) and the time you take your second measurement (posttest). The 
longer the period of time your evaluation covers, the more you need to be aware of possible 
historical effects. 



MATURATION: Maturation refers to the developmental changes that naturally occur in your 
sample subjects. Things like growing older, more experienced, or more independent can have an 
effect on your outcomes that is independent of your prevention activities. For example, if you 
are measuring attitude change in middle school students, there is evidence that social attitudes 
of seventh grade students naturally become more negative in the course of a school year. 

TESTING: Testing refers to the effects of taking a test upon the scores of a second testing. If 
you use before-and-after measurements, you must be aware of testing effects. 

INSTRUMENTATION: Instrumentation effects occur v/hen changes in a measurement instru- 
ment are made during the course of an evaluation or when observers or scorers either change or 
change their criteria for recording behaviors during an evaluation period. 

STATISTICAL REGRESSION: Statistical regression operates when groups have been selected 
on the basis of their extreme scores. When remeasured, these extreme scores tend to regress 
toward the overall group average. In prevention work, if you chose to do a program with 
students who, on a pretest survey, indicated the strongest pro-drug-use attitudes, there is a good 
likelihood that these high scores would be less extreme on a second measurement, even without 
your program's intervention. 

SELECTION: Any time the kind of people you select for your prevention group and the kind 
of people you select for your comparison/control group differ, you may be building selection bias 
into your evaluation. Random assignment is the best protection against selection effects. 

EXPERIMENTAL ATTRITION: If you have unequal dropout rates or selective dropout from 
either your prevention group or your comparison/control group, attrition may affect the validity 
of your findings. If your prevention program were studying the effects of a parent program that 
attempted to change family attitudes about drug use, and parents of students who already used 
drugs found the classes so threatening that they dropped out, attrition would threaten the validi- 
ty of your findings. 
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To get the best picture of what things would have been like without your program, in spite of 
history effects or selection problems or a lot of other things that could affect results, you should use a 
randomly assigned control group or some nearly equivalent group for comparison purposes. Deciding 
what to compaie your program with is called designing the study. 

Evaluation designs allow you to compare your results with, (1) past performance, (2) results from 
other groups having the same characteristics but experiencing different programs or no programs, or 
(3) results from groups that may differ from your prevention group in other ways. The stronger your 
evaluation design (the closer your prevention groups and comparison groups are to being identical in 
characteristics and nonprogram experiences), the more likely you'll be able to determine if your pro- 
gram made a difference. The next section reviews several of these basic designs. 



Part 1: Four types of designs that are used most frequently in the evaluation of prevention programs are 

Consider Possible covered here: 

Evaluation Designs , p. * n * n • 

^ 1. Pretest-rosttest Designs 

2. Experimental Designs 

3. Quasi-Experimental Designs 

a. Comparison Group Designs 

b. Time-Series Designs 

4. Posttest Only Designs 



Pretest'Posttest Designs 



A pretest-posttest design compares the same person or people at two points in time, before and 
after your program. The pretest-posttest design is not a strong test of the effectiveness of your pro- 
gram. For example, it does not control for other historical events that could have changed, such as 
rates of drug use in the general population. It can, however, tell you if those who participated in your 
program changed in the desired direction (i.e., if your program produces the "immediate results" you 
wanted). For example, if you had a drug abvse curriculum and you were unsure whether it was con- 
ceptually best for third or fourth grade students, you might give a pretest at both grade levels, teacn 
the curriculum, and then give a posttest at both grade le\els. By comparing the changes in drug 
knowledge, attitudes, ana mtentions to use drugs at each grade level from pretest to posttest, you 
would have more information on which to base a decision about the appropriate grade lev^el for your 
curriculum. 

v/iih a pretest-posttest design, you never know if \our program itself caused the changes you 
measure. What you use for comparibun is information about the participants before they experienced 
your program. With a pretest-posttt^t design, ^ou know how much your subjects changed on some 
measure during the period of your program. You will not know what caused the changes or if those 
who were not involved in the program might have changed as well. 



A pretest-posttest design is diagramed like this: 

0 X o 

Pretest Prevention Posttest 

Observation Service Observation 
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Use of Control and Comparison Groups 



One approach to design that can help you decide if \our program caused changes in attitude or 
behavior is the construction of comparison or control groups as equivalent as possible to your pro- 
grams participants on any variables that might influence outcomes. 

Experimental Designs 

Since it is often difficult to determine before the program which variables may be critical to natch, 
a common method for gaining equivalent comparison groups is random assignment. Random assign- 
ment can be achieved simply through flipping a coin or drawing numbers from a list tc divide par- 
ticipants into experimental or control groups. All true experimental designs employ some form of ran- 
dom assignment of participants. With ranuom assignment, you have done the best you can to 
neutralize all differences between those who receive your preventive services and those with whom 
you plan to compare them. This gives you a high degree of confidence that any differences you find 
between groups after your program were caused by your program. The participants who don^t receive 
the program are called control groups. 

Here is an example of an experiment with random assignment to different groups. The Quinn 
County Prevention Coalition has two programs that are designed to encourage a healthy lifestyle for 
young people and discourage experimentation with drugs. One program is a teacher-taught curriculum 
that takes 10 hours of instruction time over a 5-week period. The second program is a Super Stars 
program that brings 10 role models into the classroom for 1-hour presentations over a 5-week period. 
You want tu know which program has the greater impact on childrens' attitudes and behaviors. 

The sophomores at Washington High School are randomly assigned to three health classes, all 
taught by the same teacher. You pretest all students in the three classes. In Class No. 1, the teacher 
teaches the health prevention curriculum. In a second class. Super Stars are brought in for 10 one- 
hour presentations; in the third class, the control group, the teacher conducts her normal health class. 
At ihe end of the 5-week period, you give all students a posttest and compare class rates of approval 
of drug use and self-reported use to determine which program has most positively changed student 
auitudes and behaviors. The design for the program would look like this: 

Xj 0 

R-^^^^— — ►O X, 0 

^ C 0 

Random Pretest Prevention Posttest 

Assignment Service/ 

Control 

A second example of an experimental design is a delayed prevention ^.ervice design. In this model, 
every uie in your sample will receive the program, but at different tii^es. In a delayed prevention 
design, you would randomly decide who receives prevention services first and who r,^xeives the 
delayed services. One of the challenges of this design is ensuring that those who receive the preven- 
tion program first donH influence the attitudes or behaviors of those who will get the program later. 
The design may be useful if you have limited staff available to offer a program, and workshops or 
presentations must be staggered through the year for different groups or classes. A delayed prevention 
design is diagramed here. 

<0 X O 

Prevention Posttest 
Service 

O 0X0 
Random Pretest Pretest Prevention Posttest 

Assignment Service 
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Random assignment is not always easy to attain. When you work in the prevention field in schools 
or public service agencies, it may not be practical to serve one-half or less of a population while 
withholding services to a second, equally deserving group. At first glance, random assignment seems 
unfair— it denies full access to prevention services for all the population. On the other hand, if you 
do not have enough resources to serve all the students in schools, or all the PTSAs in a district, ran- 
dom assignment, the flip of a coin, may be the fairest way to decide who should get prevention serv- 
ices, especi?'' if they are in high demand. Moreover, it should be recognized that until a service has 
been proveu effective, it is not clear that random assignment denies anyone a useful service. Yet 
organizational, political, and population targeting priorities may still prevent the use of experimental 
designs with random assignment in self-evaluations of prevention programs. In situations where ran- 
dom assignment is not possible, groups can be constructed that are comparable on some but not all 
variables. These nonrandomly assigned groups are called comparison groups. The designs that use 
comparison groups are quasi-experimental designs. 

Quasi-Experimental Designs 

Comparison Group Designs. Think back to the example of the Quinn County Prevention Coor- 
dinator with two programs she wanted to test. Suppose students were already assigned to health 
classes for the year, making random assignment impossible. She could consider several quasi- 
experimental designs. The coordinator could decide to compare the effects of Super Stars and the 
health curriculum without random assignment. She might give tnree groups a 5-week program: the 
new health curriculum to one class, the 10 hours of visits from the Super Stars to another, and the 
old health curriculum to a third class. At the end of the program, she could compare the three 
groups' means on attitudes toward drug use and self-reported use. 

The design looks like this: 

X, O 
X, o 

c o 

Prevention Services/ Posttest 
Control 

Alternatively, the coordinator could have strengthened her design by comparing the three classes 
before as weli as after the prevention programs were offered. This design would look like this: 

) X, O 

O X, o 

o c o 

Pretest Prevention Services/ Posttest 

Control 

This design is much the same as the earlier experimental design, but without random assignment to 
ensure that the same kinds of students were spread equally across all three classes. 
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Perhaps you do not have access to schools or students not served by your prevention pro- 
gram. You can still use a quasi-experimental comparison group design. You may be able to com- 
pare results from your prevention project with results from the national survey of high school 
seniors conducted by Dr. Lloyd Johnston and his colleagues at the Univereify of Michigan. An- 
nual results of this survey are available by contacting NIDA. The NIDA reports provide na- 
tional and regional estimates of the incidence and prevalence of a range of drug-related attitudes 
and behaviors for a representative sample of high school seniors, which can be aned for com- 
parison purposes. For example, you may seek to reduce regular marijuana use among high 
school students through a student assistance nrogram. By comparing the rates of daily mari- 
juana use in your school with the rates reported in the annual surveys of seniors, you would be 
using a quasi-experimental comparison group design. 

The variety of quasi-experimental comparison group designs is unlimited. Observations (Os) 
and prevention strategies (Xs) can be added or subtracted in comparison group designs in any 
combination of measurements you care to create. Just remember, the more observations and 
prevention strategies you have in your design, the more complicated your evaluation project will 
be. 

Time-Series Designs. Although it is always preferable to have a comparison or control group, 
you may find that impossible. In this case, a time-series design can be considered. A time-series 
design is more powerful than a pretest-posttest design, since it allows you to observe trends in 
behavior or attitudes before and after your program. A luajor limitation of a time-series design 
is the practicality of testing your group several times before you begin your program. A time- 
series design is diagramed here: 

0 0 0 X 0 0 0 

Pretests Prevention Posttests 

Service 

Posttest'Only Designs 

If you waiit some measure of the effectiveness of your program but don't have enough par- 
ticipants, the power to construct comparison and control groups, or the opportunity to give 
pretests, you can gain some information through a posttest-only design. In posttest-cnly designs, 
you simply measure the outcomes of your program. 

A posttest-only design might be considered you see individuals or groups only once and are 
not able to administer both a pretest and a posttebt. For example, in a collaborative project with 
NBC affiliate KING 5 Broadcasting Company in Seattle, a posttest-only design was used to 
evaluate a statewide, single evening parent prevention training workshop. This evening 
workshop was scheduled 2 days after a TV docudrama highlighting the experiences of a family 
with a teenage drug abuser. Parents who showed up for the workshops (more than 4,000 on the 
same evening) were recruited through TV public service spots, the docudrama, and newspaper 
articles. These were not people who expected to be given a pretest when they showed up at a 
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parenting workshop on teenage drug abuse prevention. Therefore, the evaluation used a single-page 
posttest for participants and a followup phone call to the people who coordinated and led the 
workshop sessions. This posttest-only design is diagramed here: 

X 0 
Prevention Service Posttest 

Posttest-only designs are the weakest of the designs discussed. They can, however, provide informa 
tion that will improve your program in a nunber of areas, including targeting, marketing, recruit- 
ment, implementation, and content. While they do not allow you to compare your program s out- 
comes with any other groups or standards or with a pretest, they can be used to measure drug use 
intentions and drug use rates after a prevention initiative. 

Comparing Outcomes from Different Designs 

How these designs differ in what they can tell you about >our program's effects may be more 
easily understood by comparing graphic displays of the outcoiries from four of the designs discussed. 
Look at graphs A, B, C, a^.d D in figure 5. 

These four graphs illustrate the results of a program to prevent adolescent drug use by teaching 
parents and teachers how to teach children refusal skills. The outcome is the proportion in each 
group initiating marijuana use by the end of eighth grade after the seventh grade school and family 
prevention program. 



Fi^ure 5.— Outcomes of a School and Family Training Program in Refusal Skills 
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Pretest Pcttest 

Quasi-Experimental Design 
(Comparison Group) 



Graph A shows the results of a true expermiental design. Using this design, students were ran- 
domly assigned to classrooms with the program and classrooms without refusal skills training. Note 
that at the beginning of the seventh grade (pretest), roughly 25 percent of the students in both the 
refusal skills training group and the control group had already mitiated marijuana use, By the end of 
the eighth grade, the proportion of students who received refusal skills training and had used man 



28 

3G 



Figure 5.— (Continued) 
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juana had risen to about 35 percent, while the marijuana initiation rate had increased among their 
nontrained control peers in the same school to about 51 percent. Prevention researchers would call 
this a 16-percent reduction in marijuana initiation as a result of refusal skills training. Because 
students were randomly assigned to classrooms and had similar rates of marijuana initiation at 
pretest, prevention workers could be confident that this difference was brought about by the refusal 
skills training program. 

Graph B shows the results of a study using a quasi-experimental design. In this study, all the 
seventh grade students in one school received the refusal skills program, while seventh grade students 
from a different school without the refusal skills program were used for compariso*^ purposes. As 
graph B shows, at the beginning of the program, more students in the school that planned to offer 
refusal skills training had initiated marijuana use (about 25 percent of the entering seventh grade 
students). This would seem to be a school in need of prevention services. In the comparison school, 
only about 5 percent of the entering seventh graders had used marijuana. As graph B shows, the pro- 
portion of students initiating marijuana use over the period of evaluation was greater for the school 
with no refusal skiPs program than for the school with the program. It looks good. There was a lower 
rate of increase in marijuana experimentation in the prevention group after refusal skills training. 
What can't be determined from graph B is what caused this difference in rates of marijuana ex- 
perimentation in the two schools. It could be the refusal skills training. On the other hand, it could 
be due to the differences in rates of smoking marijuana in the two schools at the beginning of 
seventh grade. Perhaps the students in the comparison school just caught up by the end of the eighth | 
grade. The students who got prevention services may have just started smoking earlier. While the 
results look favorable, these prevention workers can't be sure that their prevention program caused 
the differences in observed rates of increase in marijuana smoking. 

Graph C shows the results of a pretest-posttest design with no comparison group. Here, the refusal 
skills training was offered to seventh grade students in one school, but no comparison group was 
studied. Graph C shows that the proportion of marijuana initiators increased in the refusal skills 
school after the program iVum 25 percent to about 35 percent. This is the same rate of increase foun ' 
in the previous examples. Was the program a success or a failure? The rate could not decrease. 
Should the prevention workers be pleased with a 10-percent increase in marijuana initiation? In the 
absence of some comparison, it is hard to know whethtr to be pleased or discouraged by these result! 
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Graph D shows the results of a time-series design. In this example, students were surveyed in fifth, 
sixth, and seventh grades before the refusal skills training program and again in eighth and ninth 
grades after the program. No comparison or control group of unserved students was studied. Using 
this design, we see that the rate of increase in marijuana initiation declined after students were ex- 
posed to the refusal skills training program. This decrease in the rate of marijuana initiation is 
desirable. The time-series design shows that the rate was increasing faster before students received the 
training. This design gives greater confidena than the pretest-posttest design in graph C that the 
refusal skills training program did have an effect on marijuana initiation. However, a concern still 
remains. What if marijuana experimentation naturally peaks among students in this community 
before seventh grade? If this is a plausible suggestion, natural maturation rather than the refusal skills 
program may be responsible for the observed results. When using time-series and other quasi- 
experimental designs, it is important to consider other possible explanations for the results observed 
and to decide how plausible they are. The more such explanations can be shown to be unlikely, the 
greater confidence prevention workers can have that their prevention program was responsible for the 
observed results. 

The design you choose can be a powerful determiner of how complete the answer is to your ques- 
tion. The closer you are able to approximate a true experimental design, the more information you 
will have to answer your question. 

For a more detailed discussion of designs, consult Campbell and Stanley's Experimental and Quasi- 
Experimental Designs for Research (1963) or similar publications. 

Part 2: 

Choose the Considering the designs discussed, which one do you think will best answer the question you're 

Design That Fits asking? 

Your Question 

and Situation 



Diagram that design in the space below, labeling the assignment process, the pretest, the post- 
test, and the prevention service. 
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You now have a question about your program, and you know what design will best answer that • 
question. How will you select the people you will serve and study? 

The people you study should be similar to the population you serve. If you are doing refusal skills 
training with sixth grade students in three schools, you should have a good representation of sixth 
grade students from the three schools in your study. 

If you are providing community workshops in setting family rules on drugs in a metropolitan area, 
you will want to include in your study a broad cross section of parents from a good portion of the 
places where you provide the workshops. Such a sample will allow you to learn how well parents are 
able to actually set and maintain rules about drugs in their families after the workshops. 

One way to ensure that the people you study resemble the people you serve is to study all the 
people you serve. In short, study the whole population served. This is called a population study. If 
the number of people served by your prevention program is relatively small, say for example 90 
children of alcoholics in support and skills training groups in three schools, you may v/ant to include 
everyone in your evaluation. 

Many prevention programs cannot afford to study all the people they serve. People who provide 
prevention curriculums to schools at every grade level may be reaching 30,000 children each year. 
When large numbers of people are served, it saves money and time to study only a portion of the 
population served. This is called a sample. 

Sampling 

How do you decide whether to study the whole population or just a sample? Three considerations 
should go i'.to making this decision: 

1. To compare groups after a prevention program, you should have a minimum of 30 people from 
each group. 

2. Consider the upper limit of your population. If you serve 100 people or fewer in the program you 
want to evaluate, study them all. 

3. Consider the cost and ease of studying all those served as opposed to a sample, given your pro- 
gram and design. (For example, when surveying students in schools, it is sometimes simpler to 
survey all students in a set of classrooms than to pull out a sample of students for surveying.) 

If you are still not sure whether to study the whole population or a sample, consult with one of 
the resources discussed in chapter 2. Usually a sampling expert can be contacted at the local univer- 
sity, at the local TV station, or at a company that conducts surveys for businesses. 

If you have decided to sample, how do you ensure that the sample you pick is similar to the people 
you serve? Loo!^ at Figure 6. Circle A and Circle B each show a sample selected from a population. 
Which sample do you think best represents the population? Circle A. How do you get such a sample? 
By picking it at random, by random sampling. Drawing people's names by chance from a hat is one 
example of random sampling. Random sampling sounds like something for experts, but actually it is 
very simple to do. 



Part 3: 

Define Your Sample 
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Figure 6.— Random and Non Random Samples from Populations. 




• = population of interest \ = sample 
Unveiling the Mysteries of Random Selection 

Obtaining a random sample of participants is practical, easy, and inexpensive. If you have decided 
that a random sample is necessary for your study, here's how you do it. 

1. Make a list of rJl the people or units (classrooms, PTAs, etc.) you serve. This is your general 
population of interest. Give each of those people or units a number, so that you have a numbered 
list. 

2. If you are sampling from a list, use a table of random numbers for the sheer ease it offers. Turn 
to Appendix C and find the table of random numbers. Close your eyes and point your finger at the 
labie If the number your finger touches matches the number you ve given to one of your program 
participants or units, that person is part of your sample. Then start down the list of random numbers. 
Each time you come to a number that matches a number on your list, assign that person to your 
sample. Continue this process until you have reached your desired sample size, lou now have created 
a random sample from a random starting point. Or. . 

3. If you are sampling from people just entering a program rather than from a list, sample every 
"nth'' person (5th person, 10th person, whatever number you choose) until you have yc t desired 
sample size. For example, to create a sample of 50 from a population of 500, you would sample every 
10th person. 

Deciding Size of Yjur Sample 

Determining the appropriate number to sample can be a complicated procedure. As mentioned 
earlier, a rough rule of thumb is that you should try to have a minimum of 30 in each of your par- 
ticipant and comparison groups. Nothing is magical about that number, however. The larger the 
samples you use to make comparisons, the less the chance that the results you find occurred just by 
chance, and the easier it is for you to demonstrate that the differences your program generated were 
statistically significant. (This simply means that they were not very likely to be the result of chance 
alone.) 

32 



40 



The sample size table in Appendix C shows you the size of two samples you would need in order 
to find a significant outcome if you were comparing proportions across two groups. For example, if 
you studied student self-reports of alcohol use during the month after participation in a refusal skills 
class and discovered that 70 percent of your prevention group had abstained from alcohol use, while 
only 40 percent of the control group had abstained from alcohol use, Appendix C shows you would 
need sample sizes of 45 in each group to be sure that this difference of 30 percent between groups 
would not occur by chance nure than 5 percent of the time. Ninety-five percent of the time you 
could be confident that the 30-percent difference you found in your sample reflected some real dif- 
feiences between the whole prevention and control groups from which you sampled. (This is the 
famous 95-percent significance level!) 

Losing ran of Your Sample (Attrition) 

Sometimes you will choose a certain number of people or units for study and then discover that 
you carnot find or collect information from some of them. People may move away, may refuse to 
complete your questionnaire, or may be absent from school or work on the day you're doing your 
study. Losing people or units from your study can affect the results. For example, research has shown 
that rates of drug use are higher among people who move a lot than among people who do not. 
Similarly, students who are absent from school on any given day are more likely to use drugs than 
students who aren't. You can do two things to be sure that your study isn't compromised by loss of 
the sample. First, select a 15- to 20-percent larger sample for your study than you will ultimately 
need. If people are lost, the sample will still be sufficiently large. Second, take precautions not to lose 
those people you have sampled for your study. 



STAFF/TEAM AGENDA 2 

ATTENDING: For a small program, the whole staff. For a larger program, repicsentatives who have 
an interest, some expertise, or a critical role in the success of an evaluation. If someone has identified 
an evaluation resource, e.g., a student of research designs or evaluations, you may want to invite that 
person to assist you with questions of evaluation design. 

TIME: Schedule 2 hours. 

OBJECTIVE: To choose an evaluation design after comparing pre-post, experimental, quasi- 
experimental, and post-test only designs with your program questions and resources. 

L Discuss the possibility of control or comparison groups. (Use chapter 4 as a reference. It would be 
a good plan to have key people read chapter 4 before the meeting.) 

IL Consider control, comparison, pretest, posttest, and time-series designs. 

III. Choose an appropriate design. 

IV. Ask your planning staff to read chapter 5 before your next meeting. 
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Chapter 5 



Outcome Measurement 

Part 1: Step 3: Designing Measurement Instruments 

Measuring the 

Results of Your Drug abuse prevention aims to produce two broad categories of effects, immediate results and 

Program longer term outcomes. Reduction in drug use or abuse is, b> definition, the ultimate goal of preven- 

tion programs. Measures of drug use provide information on the long-term outcome of ultimate in- 
terest for prevention. 

For that reason, it makes sense to try to measure changes in participants' intentions toward and ac- 
tual use of alcohol and/or other drugs whenever possible. In prevention activities, however, immediate 
results such as changes in attitudes towards drugs also may be important to measure. Prevention pro- 
grams focus on changing behavior at a future point in time, and this distance from the time of pro- 
gram involvement can sometimes preclude measurement of actual substance abu5>e outcomes for a 
considerable period. (For example, behavior change resultmg from a drug education program in 
elementary school may not be measurable for several years, but intentions regarding future drug use 
can be measured as an immediate result of such a program.) 

Measures of Outcome 

Different prevention programs may seek different prevention goals. A program targeting fifth and 
sixth grade students may seek to prevent or, at minimum, delay the age of first experimentation with 
tobacco, alcohol, and/or marijuana. Alternatively, prevention efforts on a college campus may seek to 
reduce the regular use of tobacco or alcohol. A prevention program in the workplace may seek to 
target alcohol or other drug abuse among employees. In deciding what drug use measures to use in 
assessing your program's long-term outcomes, be clear on what level and type of drug-using behavior 
you seek to prevent and use or construct measures of that behavior. Appendix B contains examples 
of items measuring a range of drug-using behaviors that you may choose to use. An advantage of us- 
ing items selected from Appendix B is that these items have been used successfully in previous 
studies. We know that they work. Further, for the items in Appendix B, national averages or 
**norms'' have been established. If you use these items, you can compare the results of your preven- 
tion work with national norms. 

Sometimes, the effect of your prevention program on actual drug use may not be visible for a long 
time. As noted above, a drug abuse prevention curriculum offered in primary school may not affect 
actual use of drugs Mke tobacco, alcohol, or marijuana for several years. In such instances, other 
measures related to drug use may be used to assess immediate results. These include: 

1. Intentions to use drugs. Do people served by your program expect to use drugs in the future? 
An example question might be: "Do you think youli drink alcohol when you're an adult?'' 

2. Attitudes toward the use of drugs. Do people served by your program approve or disapprove of 
the use of drugs? An example item is: "Do you think it is okay for someone your age to smoke mari- 
juana?" 

3. Perceptions of risks associated with the use of drugs. Do people served by your program under- 
stand the health risks or negative social or other consequences associated with drug use? An example 
question is: "Do you think it hurts people if they smoke marijuana regularly?" 

By measuring intentions, attitudes, and perceived risks of drug use, you can learn whether par- 
ticipation in your program v> associated with immediate positive changes in variables that have been 
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shown to be related to drug use itself. You can assess your program's effectiveness in changing inten- 
tions, attitudes, and perceptions without waiting to measure actual drug use. 

Drug use itself can be measured in at least four different ways: (1) by the self-reports of people you 
are studying; (2) by using biochemical assay procedures on samples of saliva, urine, or blood provided 
by people you are studying; (3) through parent reports, police reports, school records, or other institu- 
tional records; and (4) by direct observation. 

Of these four measures, the most frequently i sed in prevention research are self-reports and reports 
of others. Although one might think that self-report of a potentially incriminating activity as drug use 
might be of questionable validity, studies have shown that self-report data provide reliable and valid 
estimates of drug-related behavior (Marlatt et al. 1986; Elliot and Huizinga 1984). 

The validity of self-reported drug use can be increased by using biochemical assay procedures 
(Pehacek et al. 1979; Luepker et al. 1981). Although both self-reports and biochemical assays are used 
by researchers in the treatment and prevention fields, biochemical techniques may be beyond the 
capacity of most prevention programs committed to self-evaluation. Observations, though valuable as 
an outcome measure for high-frequency behaviors, are difficult to obtain for hidden behaviors such as 
illicit drug or alcohol use. Self-report and record data are often adequate to provide information on 
drug use. 

Measuring Immediate Results 

Although prevention programs ultimately seek to affect dri'g use outcomes, sometimes the im- 
mediate results being sought are in other areas. Look back at the model of your program that you 
drew in chapter 3. What immediate results did you specify for your prevention program in that 
model? Perhaps you hope to improve parents' family management skills or to increase childrens' 
knowledge of the social or health risks associated with the use of drugs. It is important to measure 
your success in achieving this immedi::te result for two reasons. First, information on the immediate 
results of your program provides early -vidence of effectiveness, e en in the absence of information 
on long-term drug outcomes. Second, if you have not produced the immediate results you seek, you 
may have little reason to expect that long-term drug outcomes will be affected. Measuring immediate 
results allows you to know whether the program needs to be altered or improved before you conduct 
a longer term study of drug use effects. 

Let's look at an example of a program model and how measures of immediate results and drug use 
outcomes can be gleaned from the model. The social development model of adolescent drug use 
(figure 7) is used in the authors' prevention work. Prevention programs based on that model seek to 
increase young people's opportunities for prosocial involvement, their skills for success in those in- 
volvements, and the /ewards they receive for positive participation with family, school, and peers. 
Parental drug use is also addressed along with parents' attitudes toward their children's use of drugs. 
Prevention activities based on that model include working with teachers to improve instructional skills 
and classroom management, working with families to clarify family rules about drugs and ensure con- 
tributing roles in the famil> for all members, and providing opportunities for positive peer interaction 
and influence in school classrooms. This work is expected to yield a number of immediate results— 
namely, strengthened social bonds to family and school, increased attachment of children to their 
parents and teach.. , increased commitment to their families and to educational pursuits, and 
strengthened belief in the moral order. This last result is expected to inhibit violations of that order, 
such as the illegal use of drugs by adolescents. 
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Figure 7.— Social Development Model of Adolescent Drug Use 
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Evaluating this work involves mtasuring the followinj, immediate results of these programs: 

■ increased social skills 

■ Perceived rewards for :n.et*aolion with prosocial children and adults 
5 Pr rewards for ii.volvement m legal activities 

■ /• nt to teachers, parents and peers 

■ Commitment to school 

■ Belief in the moral order 

As you look at your own program model, identify the elements that you expect to change mi 
mediately as a result of your prevention work. List these, "'hese are the immediate results you should 
seek to measure in your self-evaluation. 

Many instruments for measuring the effects of drug prevention programs have already been 
developed and tested, and adopting or adapting an existing measure is suggeste^.^ whenever possible. 
Such an instrument will have proven reliability and validity and can reduce some of the error in your 
findings. You can also borrow items or groups of items from different instrumen .s to build an instru- 
ment that addresses the questions you want answered. An instrument you migh* consider adopting or 
adapting for your evaluation study is included in Appendix B of this handbook. 

Whether you use a previously tested measure or your own instrument, you should always field test 
it before beginning the formal collection of data to make sure that it will meet the needs of your 
study. For example, you would most likely need to simplify or modify an instrument designed for 
high school students, before administermg it to fifth graders. 
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To field-test an instrument, ask people with characteristics similar to the group you plan to study to 
complete it. For example, if you are developing a questionnaire for students, you may want to ask 
children of friends to fill out your draft questionnaire. Once they have completed it, review the ques- 
tionnaire with them. Ask if any questions were unclear or confusing. Be sure that they understood 
the possible responses. Also keep track of how long it took to complete the questionnaire so you can 
know if It IS too long to be given in the time allowed. Ask if the field test respondents became 
fatigued or bored while completing the questionnaire. Based on this field test, you can revise the ques- 
tionnaire to fit the study sample. The more field testing you do, the better your final instrument will 
be. 

Operationaiize Your Question 

The effectiveness question you developed in chapter 3 will guide you in choosing what to measure 
and how to measure it. 



Write your question once again. 



*** THE QUESTION 



* 

* 

* 

* 

* 

* 

* 

What kinds of behavior is your question asking about? How will you recognize your outcomes when 
you see them? 

Consider, for example, this question raised about a parent training program: "Does parent training 
m settmg rules and consequences for drug use increase childrens^ perceptions of risks associated with 
drug use and in this way delay drug initiation?" 

The two outcome measures of interest in this question are an immediate result, "perceptions of 
risks, ano a longer term outcome, "childrens' initiation of drug use." How would you measure these 
two variables? 

The first step is to operationaiize the variable. In other words, define the variables so you know 
what constitutes "perception of risk" and what is meant by "initiation of drug use." 

■ Perception of risk could be defined as a child's report that there would be clear 
and explicit negative family consequences if he or she experiments with drugs. 
This could be measured using student survey items. 

■ Likewise, childrens' initiation of drug use could be defined as any experimentation 
with alcohol/drugs and could be reporteo by self-report survey quest-ons. 

If the outcomes of the parent training question were operationalized, the question might read like 
this: "Does parent training in setting rules and consequences for drug use increase perceptions of 
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risks, as measured by children's self-reports of perceived risks of drug use, and delay children's initia- 
tion of use of alcohol and other drugs, as reported by self-report data on drug use." 



As you operationalize your outcome variables related to drug use, refer back to the first pages of 
this chapter. Do you expect your prevention work to reduce problems associated with any drug use, 
regular drug use, or to prevent drug initiation, or the use of a specific drug? For each relevant out- 
come, select or design an appropriate measure. If you seek to prevent problems associated with drug 
use, you will need to measure bioth the frequency and duration of use and assess, for each person in 
your sample, whether evidence of impairment in functioning related to drug use is apparent. If you 
seek to prevent regular use of drugs, at a minimum you will need to measure, for each drug of con- 
cern, the number of times the drug was used or the frequency of use over a specified period. Most 
prevention researchers ask about the frequency of use in a 30-day or 1 -month period. You may also 
want to measure the quantity or amount used for certain drugs, such as alcohol. If you seek to pre- 
vent drug initiation, you will need to ask whether each person in your sample has ever tried each of 
ihe drugs of concern in your study. All these measures can be designed to focus on the initiation and 
use of any specific drug of concern. 

Now, operatiOnalize your question so you will know what specific attitudes, behaviors or 
characteristics you are going to measure as outcomes of your evaluation. (You may write in the hand- 
book or use the evaluation think sheet in Appendix A.) 



Write "THE QUESTION" here: 



List the long-term outcome variables in your 
question 



List specific measures of the variables 



List the immediate results in your question 



specific measures of the immediate 

, wSUltS 



Rewrite your question with the immediate and longer term outcome variables operationalized: 



ERLC 



38 



4( 



G 



Once you have operat.onalized the variables, you will need to collect data in o.der to answer your 
questions regarding immediate results and long-term outcomes. Consult the data collection instrument 
in Appendix B. Look for items that measure the variables that you need to measure, and use or adapt 
those Items. If you cannot find items that measure the variables of interest to you. you can consult 
with a resource person about existing items that may measure what you want or develop your own 



Staff/team agenda 3 is included at the end of this chapter to help you and your staff/team col- 
laboratively plan how to measure the outcomes of your program. 

n!lL f ''"0* wl^^^ outcomes your study will be assessing, you're ready to decide what other in- 

RoT A . formation you need about the subjects of the study. What are the characteristics of your sample? 

Background Data Some background characteristics commonly measured include: age, sex, race, presenting problems 
(substance use history if any), socioeconomic status, cultural background, family composition, and 
family history of alcohol or drug use. The background information you collect for your study will de- 
pend on the characteristics of your sample and the evaluation questions you are asking. 

List the background information you will need • able to talk about "who" the people were who 
I^rticipated in your study. What cnaracteristics of the.e people do you think will be important given 
the questions you ask? You will want to list both the background information you need to collect and 
the reason you think it is important. 



Background Information Item Reason for Collecting It 



As you begin a STEPP evaluation, your enthusiasm may encourage you to try to collect all kinds 
of information rather than only the data you need to answer your question. All information is in- 
teresting, but only some of it is essential for you to answer your evaluation question. Just collect the 
background data you need! Keep it simple, from the design through the data analysis, and you will 
have both usable data and a staff that will look forward to the next evaluation process, curious to 
learn more. 
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STAFF/TEAM AGENDA 3 



OUTCOME MEASUREMENT 

ATTENDING* Staff members who have a particular interest in or understanding of program out- 
comes, including program or clerical staff who will followup with participants after prevention 
programs. 

TIME: Schedule 2 hours. (If you decide to design your own instruments, you may need to delegate 
tasks and return again. You may want to seriously consider customizing the pretested instrument that 
you will find in Appendix B of this handbook.) 

OBJECTIVE: To consider alternative instruments for outcome measurement and choose ones that are 
appropriate for the questions you are evaluating. If instruments are not available to serve your needs, 
to design an instrument that will fit your evaluation questions. 

I. Review the instrument that is presented in Appendix B. Remember, this instrument may be 
modified to fit your program's needs. Gather other instruments which may measure variables you 
think are important. In considering instruments, you must be as sensitive to the ease of administra- 
tion as to the usefulness of the data. (Chapter 5 discusses choice of measures and creation of your 
own measures.) 

II. Choose a measure or measures that will answer each of youi effectiveness questions. If you design 
an instrument, keep in mind a format that will allow for easy transfer of the information to a com- 
puter or data sheets for data analysis. 

III. Devise a plan to field-test your instruments for suitability for your population. 
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Chapter 6 

Implementation (Effort) xMeasurement 

Implementation or effort measurement answers the question "What actually happened in the 
prevention program?" Implementation or effort measures tell you about: (1) level of effort (2) level of 
participation, and (3) quality of program delivery. 

Effort measures can tell you how many parent classes were offered, how many lessons were taught 
by teachers using the prevention curriculum, how many times staff made presentations to community 
groups, how many parents actually attended parenting classes, or how many phone calls were re- 
ceived on an alcohol abuse prevention hotline. 

Effort measures also can provide information on how closely the actual program followed the initial 
design. If teachers promised to teach three alcohol prevention lessons a month, how many were ac- 
mally taught? Did teachers actually use the interactive problem-solving strategies they were taught'' 
How many of the parents in ihe parenting classes applied the skills they were taught when they got 
nome. Wi.ai acti;any x,appens when someone from the student assistance program counsels a peer 
about drug abuse? 

Not all effectiveness questions require that you ask implementation questions to arrive at an 
answer. If you are asking an effectivene.ss question, such as "Did our seventh grade refusal skills cur- 
riculum reduce the self-reported use of alcohol among students in the prevention class?," it is possible 
to achieve the answer without ever asking "Did teachers actually use the curriculum as we taught 
them to? How many classes did each teacher teach?" or "Did the teachers use direct instruction or 
atTain Ss' '^^^'^ ^^e program and tell exactly what was done to 

If you do not measure the effort that was undertaken to bring about the desired outcomes 
however, your program will remain a "black box" in your evaluation. If your program fails to achieve 
the desired drug use reduction outcomes, you will be unable to tell whether that was because the pro- 
gram was not well implemented or because the program, though well implemented, didn't work to 
preven drug abuse as expected. Alternatively, if your program succeeds in preventing drug use, you 
will not know what it was in your program that produced these effects. 

For example, the Quinn County Prevention Coalition asked a local school district to collaborate in 
evaluating the intermediate (grades 3-5) component of its drug abuse prevention curriculum. Two 
schools of equal size with similar student characteristics were chosen to be the experimental and con- 
trol schools for the evaluation. 

All teachers at the experimental school were given copies of the drug abuse prevention curriculum 
and asked to teach 12 of the prevention lessons for their grade level in the first 3 months of the 

T yf/,Teachers af the control school were asked to follow their normal teaching patterns. All 
hird to fifth graae s'.udei.ts at both schools were given a survey about attitudes toward drug use at 
the beginning of the project and at the end of the 3-month training period. 

When the data from the pretests and posttests were analyzed, it was found that: 

1 There were no significant differences between third grade students at the experimental and con- 
trol schools on attitudes toward drug use. 

2. Fourth grade students in the experimental school had more negative attitudes toward drug use 
than fourth graders at the control school, but the outcc.-. is were only marginally significant. 
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Part 3: 
Consider 
Instruments 
for Measuring 
Impleme tation 



3. Fifth graders who had been exposed to the drug abuse prevention curriculum had attitudes that 
were significantly healthier than both drug use attitudes of their fifth grade control group peers and 
the drug use attitudes of the third and fourth grade experimental studerts in their school. 

How could one curriculum produce such mixed results at three different grade levels'' One obvious 
possibility is that fifth grade teachers actually used the curriculum more than third and fourth grade 
teachers in the same school. 

Without measures of implementation (i.e., information on the number and type of lessons actually 
taught by third, fourth, and fifth grade teachers, observations of teaching strategies used by both con- 
trol and experimental teachers, and knowledge of drug-related evenis at the two schools during the 
3-month test period), it would be almost impossible to explain why the results differed at each grade 
ievel. 

Implementation measurement aiso allows you to assess differential effects of your prevention pro- 
gram. For example, implementation measures of a parenting program will allow you to answer these 
questions: Is the program more effective with parents who attend all sessions than with parents who 
attend only a portion of the sessions? Is the program more effective for families with younger 
children or families with older children? 

"Critical mass" may a factor in the success c: prevention activities. What we do to prevent 
substance abuse may be a less powerful factor than "how much" we do of it. Wiih-^-^t measures of 
implementation, sve can't determine how much is enough to get results. 

It is true that measuring "what we do" often results in more paperwork for your program staff. 
However, implementation measj^*?., already exist in programs that have a management information 
system in place. Whether or you're involved in evaluation of your program, you may need im- 
plementation information to know what services your prevention program is actually delivering. 
Although program staff may initially question the value of turning in monitoring forms showing how 
they've used their time, this monitoring of effort actually documents their value to the program. 

Most projects either use existing information sources to measure implementation or design their 
own instrument^ for this purpose. The first step in designing your implementation instrument is to 
decide what questions you want to ansvver with your monitoring system. Here are some implementa 
tion questions that previous evaluations have addressed: 

1 . How many days a week did each teach«:r use cooperative groups in teaching the refusal skills 
curriculum? 

2. What other drug abuse programs were operating in the school/community, concurrent with our 
prevention curriculum? 

3. Who (by categories) attended our community drug awareness presentations? 

4. From what social groups were our 10th grade student peer counselors drawn? 

It is important to have ^ use for any category of information you choose to collect. (Don't forget: 
fCeep it simple!) If specific uses are communicated to your staff, you are more likely to attain full par- 
ticipation in your monitoring program. Listed next are the uses of the data to be collected in ques- 
tions 1-4. 
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1. Since cooperative classroom structures have been shown to expand friendships and provide in- 
creased .academic success, teachers' use of cooperative techniques may lead to both positive peer at- 
tachments and increased commitment to school, factors that could decrease the chance of a youne 
person s experimenting with alcohol and drugs. 

2. One possible explanation of surprising results might be the presence of other programs that were 
affecting either prevention or comparison groups. 

3. If you know the types of people attending community drug awareness presentations, you can 
tailor presentations to their needs. You will also know which target populations you're missing. 

4. When you look at the impact of your peer counseling program, you will want to ask why some 
s udents report that peer counseling positively affects their drug use and others report no difference 
It may be that the individuals not affected by the program were from groups not chosen to be peer 
counselors. ^ 

Measures of Time Used 

Use of time is one of the most common implementation measures. Measures of time should be 
categorized into standard intervals with some way to distinguish among different types of staff ac- 
tivities The time intervals to be used to record activities of yo-u staff or team must be decided. Time 
■ntervals small enough to pick up different activities, but not so small that they become a paperwork 
burden, are required. Categories of activities to be monitored also must be chosen. Examples of three 
measures are shown in figures 8 and 9. 

People frequently fail to make a logical linkage between the question to be answered and the 
measures used. However, if you follow the strategy demonstrated, of first specifying the questions you 
wan your instrument to answer and then listing why you need the information, you should know 
what measures to use from your existing management information system or be able to design an in- 
strument that will give you the accurate and useful implementation information that you need. 

inc?ude: "^'"^ '° ''"'^"'"^ instruments to measure implementation 

■ Do our needs require a staff-activity or a client-centered approach? 

« When and how frequently will data be collected? Do we want to record informa- 
tion every day or sample units of time for measurement? 

■ How valid and reliable is the information likely to be? 

■ How will information be fed back to the staff who have collected it? 
rmtr!!;""^ r" "'l^' ^'''"'''^ '^''P''' demand consideration by all who 
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Figure 8.— Instrumei.. to Measure Program Effort 
MEASURING PROGRAM EFFORT 



Staff Name Week of 
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Aciiviiy 


Planning 
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Notes and 
Explana- 
tions 



















































PREVENTION 

ACTIVITY CODES: TIME CODES: 

1. Peer Counseling ! = 15 minutes or less 

2. Teacher Training 2 = 15-30 minutes 

3. Parenting Training 3 = 30-45 minutes 

4. School Advocacy 4 45 minutes to one hour. etc. 

5. Other 

Suuac Austin, Mic^hacl J el al fSialuaii/i^ )our Aj^encK s Programs Cup>nght 1982 b> Sage Publications, Inc Reprinted by 
|)crmission of Sage Publications. Inc 
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Figure 9.— Instruments to Measure Staff Activity 



PREVENTION SERVICES 
STAFF ACTIVITY FORM 

Staff Name Week of 





DIRECT SERVICES 


INDIRECT SER VICES A DVOCACY 


Organization 
Contacted 


Planning 
Time 


Service 
Time 


Delivery 
ft Served 


Planning 
Time 


Service 
Time 


' Delivery 
ft Served 


Planning 
Time 


Activity 


COL. 1 


COL. 2 


COL. 3 


COL. 4 


COL. 5 


COL. 6 


COL. 7 


COL. 8 


COL 9 



PREVENTION SERVICES 
GOAL-FOCUSED ACTIVITY FORM 



ORGANIZATION 
CONTACTED 


TYPE OF 
CONTACT* 


GOAL I: 
To increase work 
opportunities for youth 


GOAL 2: 
To increase abilities 
of community members to 
cope with stress 


GOAL 3: 
Development 
Activities 






Time 


tf Served 


Time 


ft Served 


Time 



•CONTACT CODE 

1 = Direct Service 5 = Receive training 

2 = Provide consuliation (indirect services) 6 = Advocacy activity 

3 = Provide training (indirect services) 7 = Planning, general information, sharing contact 

4 = Receive consultation 8 = In-house planning or preparation 

Source Austin, Michael J el dl Exaluatm^ Your Agenc\ \ Programs Copyright 1982 by Sage Publications. Inc. Reprinted by 
permission of Sage Publications, Inc 
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Designing Implementation Instruments 

Implementation QiJestions I Need to Answer 
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3. 
4. 



I. 

2. 

3. 
4. 



How Will I Use This Information? 



Who Should Be Involved in the Design? 



First Draft of My Form 
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STAFF/TEAM AGENDA 4 
IMPLEMENTATION MEASUREMENT 



ATTENDING: Supervisory Staff and Representative Service Providers. 
TIME: Schedule 2 hours. 

OBJECTIVE: To consider alternative instruments for measuring effort, by reviewing existing in- 
struments and considering the costs and benefits of creating your own instruments. 

I. Review available instruments as well as forms that are currently in use within your organization 
that measure program effort. Consider the time and energy costs of the staff members who will be 
responsible for this data collection. 

II. Choose or design instruments to measure .ur effort questions. 

III. Devise a plan to test the validity, reliability, and practicality of your instruments for your 
program. 
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Organizing and Collecting Data 

Step 4: Building a Data Collection Plan 

Part 1: In chapter 5, you operationalized your eva. iation question so you know what information you 

Plan How to need to collect. The next step is to build the plan that shows how this work is going to get done. 

Collect Data 

As discussed in chapter 5, three primary sources of outcome information are available: self-report 
data, records or ar:hival data, and observational data The questions you ask will determine which 
sources you choose to tap. A brief description of the strengths and limitations of each data type is in- 
cluded here. 

Self-Report Data 

Self-report data can provide information on attitudes, beliefs, behavior, and attributes. It can use 
either an open-ended question format: 

"How much did you like this class?" 



or a closed-ended question format: 



"How useful was the presentation on family bonding?" 
Useful 1 2 3 4 5 Not at all useful 



An open-ended format will probably give you more information but will be n\vch more difficult to 
summarize. Closed-ended questions are easy to analyze For example, by calculating the mean score 
on the above scale, you can obtain a summary indication of participants' views. Open-ended questions 
will help you interpret your findi.igs. You may want to use both kinds of questions. 

Interview^. Interviews involve an interviewer's asking someone questions, creating an interactive 
situation that is particularly advantageous if youVe concerned about a subject's abiluy to read or 
write. Interviews also allow you to stimulate a response or to extend a particular line of questioning. 
Intel v)ews may be unstructured or open-ended, allowing respondents to talk freely about their own 
experr^nces, or they may be structured, presenting a fixed schedule of questions to be answered, 
son.wtimes using fixed response choices. Unstructured interviews are useful when you are trying to get 
a feel for issues or emerging events or experiences. Structured interviews allow responses to be com- 
pared and summarized across respondents. With interviews, you can customize your instrument so 
that not all respondents need respond to all questions. You should expect higher response rates with 
either face-to-face or telephone interviews than you will achieve with questionnaire data. 

Face-to-face interviews will give you a higher response rate than telephone interviews, but the cost 
of collecting data in face-to-face interviews is high. There is also the risk with face-to-face interviews 
that the hiterviewer can bias the information collected. On the other hand, confidentiality and privacy 
may be more difficult to maintain in phone interviews. The presence of other family members may 
affect answers on the phone. 

Questionnaires. Questionnaires involve the completion of a pencil-and-paper data collection instru- 
ment by the subject and are usually the cheapest method of obtaining self-report data. Questionnaires 
can be administered in person to individuals or groups, or they can be done by mail. The response 
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rate for in-person questionnaires will be considerably higher than response rates for mailed question- 
naires. If yon are administering in-person questionnaires to your sample, you can anticipate a comple- 
tion rate of 85 percent or higher, depending on the population studied. If you administer your survey 
by mail, a response rate of 70 percent is considered very good. If you have less than a 50 percent 
response rate, your data may be too incomplete to be usable. 

Return rates of mailed questionnaires are subject to selective bias that can affect your findings. 
Those who liked your program are the most likely to return questionnaires. Also, those with a higher 
educational background are more likely to take ihe time to wade through written questions and in- 
structions. Frequently, information is needed from people who are least inclined (or able) to complete 
a mailed survey form. 

A combination of mailed and telephone survey techniques is sometimes used to maximize response 
rates at the lowest cost. You can do this by sending a survey by mail to all those in your sample. 
After 2 weeks, you should follow up with a reminder card to those who have not returned the 
original survey. This procedure should result in responses from 35 percent to 50 percent of your sam- 
ple. You can then call those who did not respond by mail and ask them to complete the survey by 
phone. The use of this phone followup should raise your rate of completion to over 60 percent of the 
sample. 

Self-report measures were designed to avoid the problems of data gathered strictly from official 
records: record data understate the frequency of events and are subject to institutional bias. Self- 
rpports are probably as good a measure of behavior as can be obtained, short or observation. An ex- 
cellent source for constructing your own self-report instrument is Dillr an's Mail and Telephone 
Surveys (1978). 

Records or Archival Data 

Archival data can be found in your agency or program records, school records, err .oyer records, 
other agency records, or police and court records. The fact that the information has already been col- 
lected makes archival data relatively inexpensive to obtain. Since other people collected the data for 
other pur, -^ses, however, you can never be certain what biases they may contain. If you do approach 
other organizations for access to record data, you must be very specific in describing what informa- 
tion you need and should plan to work collaboratively with the organization to obtain access in the 
manner least disruptive and most compatible with the organization's policies and procedures. Gaining 
access may require several months before the transfer of record data is completed and data a^e 
available for analysis. If you hope to use official record data, begin your negotiations long enough in 
advance to ensure that you will have the information when you need it. If you plan to collect infor- 
mation from schools, see Gaining and Maintaining School Access for Your Evaluation. 
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Gaining and Maintaining School Access for Your Evaluation 



If your evaluation project requires that you collect information from students in schools or use 
student records of achievement, disciplinary actions, or instances of reported drug use, you will 
need formal permission from your school district to do your evaluation. Many school districts 
have a formal application procedure that must be followed in order to collect data from or on 
students. This application requires a detailed program description and a description of your 
evaluation. It is important to do a good job in completing this application, and you may want 
to ask a resource to help with it. Draft your application, and then take it to a professor in 
education or the social sciences you know and ask him or her to edit it or review it for ^ou. 
Perhaps the husband or wife of a staff/team member can help with this editing. 

If you want to maximize the chances tiiat you are able to do a successful evaluation, you may 
wish to consider the following additional steps in gaining access to your school district. 

1. Convince the district decisionmakers of the importance of your prevention program. You 
may want to present the program to the superintendent of schools and/or the school board. You 
can increase the chances that they will be able to focus on the importance of your prevention 
program if you include in this first session a trusted community leader who endorses your pro- 
gram and a member of the superinte;ndent's cabinet who is also convinced of the importance of 
the prevention work you are doing with the district. Explain simply and clearly to the decision- 
maker what you are seeking to do in the school district and why it is important to answer your 
evaluation question. 

2. Another approach is to ask the district to ^nvene an ad hoc committee of representatives of 
the different groups involved in the district: the school board; central administration; building 
level personnel including principals, teachers, and representative^ of the PTSA; and other groups 
involved in the district who may be affected by you: prevention program. Present your question 
for evaluation to this group Tell them that you are mterested in understanding the effectiveness 
of your program in preventing drug abuse among school stud'^nts. Explain the purpose of your 
study clearly, and you should find people eager to help you get the information you need 
without compromising the privacy or rights of any party. 

In short, it is important to recognize that the formal research application process may be only 
part of what you need to do to ensure thai you're able to successfully evaluate your program. If 
it is important to gain access to the schools, it is important to take the time to think thror5;h 
the best way to do it. Once you have gained access, maintain it by reporting information from 
your project back to your school collaborators in a timely manner and in a way that makes it 
useful to the school district. 



Observational Data 

Observational data provide the most valid and most expensive measure of behavior. It is rare- 
ly possible to observe people 24 hours a day, however, so it is essential to design observation 
periods and observational instruments that give a representative sample of human behavior. It is 
also important to design observational systems that are not intrusive, so the behavior you 
measure will be as ciose to natural as possible. 
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Observational instruments exist for just about any behavior you have an interest in measuring. 
They can be grouped into three types: checklist instruments, interactional instruments, and 
naturalistic or open-ended instruments. 

Checklist instruments predefine the behaviors that the observer is looking for, and they allow for a 
quick checking off of behaviors whenever they happen. Checklists usu ally ask that the observer check 
a behavior category or categories at regular time intervals, such as every 30 seconds, 1 minute, 5 
minutes, etc. Checklists are relatively easy to complete. They are selective instruments that look at a 
reduced range of human behaviors. If you know what you want to measure, a checklist can be an ef- 
ficient way of measuring. For example, if you are concerned about student drug use in the restrooms 
of your school, you might develop a checklist to be used tw-^e each day in monitoring restrooms. The 
checklist could include categories to note whether any evidence of drug use (i.e., matches or parapher- 
nalia, odor of tobacco or marijuana, groups of students) was found during each monitoring visit. 
Figure 10 shows an observation checklist used to measure teaching behaviors in school classrooms in 
a prevention field experiment. 



Figure 10.— Classroom Observation Instrument 
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Interactional instruments measure the dynamic of human interaction. They attempt not only to 
note the predominant behavior of a timed moment but to record the pattern of interaction — who 
talked to whom. An interactional instrument is appropriate for certain pre\ention-relevant tasks. For 
example, if you want to increase friendship attachments across cliques in a middle school in hopes of 
changing peer influence networks, you may want to measure interactions among students in 
classrooms or in a cafeteria. The instrument shown in figure 10 also measures student interactions in 
the classroom. It is used to see how teacher behavior affects students' classroom behavior. 

Unstructured or naturalistic observations provide the most comprehensive representation of 
behavior. Since they are not limited by predefined categories of behavior, they record whatever the 
observer notices. The cost of obtaining a "camera view" of life is the difficulty in coding the data for 
analysis and difficulty in obtaining reliability among observers. The benefit is a participant's view of 
what is happening. For example, just by keeping an eye on the high school parking lot during the 
weeks before and after a "steer clear" driving safety campaign in the school, you may be able to 
observe changes in students' behaviors around and in cars that suggest effects of the "steer clear" 
campaign. Perhaps you will find fewer beer cans in the parking lot a*. ^nd of the day or see fewer 
students driving their cars recklessly at lunch time. 

If you choose to use an observational instrument as part of your evaluation plan, you must build 
in time to train your observers, in order to obtain consistent and accurate information across 
observers. 

If structured observation is used, categories must be defined so that each observer knows the 
specific behaviors that should be coded in that category. Also, observers' responses must be compared 
to build your confidence in the reliability of the data across observers. 

Training people to use observational coding systems is time-consuming, but the technology is readi- 
ly available. You may want to hire a social work or psychology student to help in developing a 
coding system and training your staff ?.s observers if you decide to collect observaiiOhdl data. 

Confidentiality and Consent 

The confidentiality of individuals must be maint^jned in any evaluation. It is important to collect, 
store, and report information so that no ir»formauon about particular individuals is released. You will 
find thai taking the time to design instruments that protect people's identity, and establishing pro- 
cedures that keep individual data from being reported, will ensure a higher response rate from the 
people in your study. Use numbers rather than names to identify individual subjects. 

If you think there is any risk to individuals or of individuals' being identified in your study, you 
need to obtain a signed, informed consent from the subjects of your study. If you are collecting infor- 
mation from children, you ould get their parents' consent as well. Such a consent form should 
describe the study, the possible risks, the individual's ability to withdraw from the study at any time, 
an ' the procedures you use that safeguard individual confidentiality or anonymity. In addition, you 
need to provide the individual an opportunity to ask questions about your evaluation either in person 
or by telephone, and ask that they sign a consent form for your records. A sample consent form is in- 
cluded on the next page. 

How Often to Collect Outcome Data 

Three important points for collecting information related to program effectiveness are. before the 
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FARIiNT COP^' 



LnivcrsiiN of Washington 
SEATTLE SOCIAL DEVELOPMENT PROJECl 
Parent Questionnaire 
Consent Eorm 



This questionndire is being sent to parents whose children attend fifth or sixth grade classes in Seattle 
schools participating in the Seattle Social Development Project The purpose of the Project is to find 
ways 10 make school a more positive experience for both parents and children, and to increase 
students* commitments to educational goals As part of the Project, we need to learn about parents' 
opinions and attitudes concerning child rearing. E\er\one agrees that raising children is no easv task, 
but people often disagree on how to do it Your participation ni the survey will help us to learn what 
parents in Seattle think about those issues Some of >ou may remember receiving a similar question- 
naire several years ago. The survey is being done again this year in order to ^e what changes have 
occurred now that the ci Jren are older. For others, this will be the first time you have been asked 
to complete this questionnaire 

E\er\ thing \ou answer in this questionnaire is confideniial Your answers will be put together with 
those of oiher parents so that researchers can count the number of times all people answered a ques- 
tion a certain way VVc would also hke to compare parents' opinions with students' opinions The in 
formation will be kept by the Project staff at the University of Washington for the entire time the 
siud\ IS conducted Your name will never be attached to project results Access to the information is 
Imiited to the staff of the Seattle Social Development Project 

The questionnaire should take approximately 15 minutes to complete We would like \ou to answer 
all of the questions However, if there is an> question that you find objectionable for any reason, just 
leave it blank Participation in the survey is entirely voluntary There will be no pcnalt\ or loss of 
benefits to which you or your son or daughter is otherwise entitled should >ou decide not to par- 
ticipate If \ou agree to participate, please {yrmt and sign >our full name below 

WE WOULD LIKE TO SEND YOU S4 00 FOR COMPLETING THE QUESTIONAIRE 
Therefore, when \ou have finished the questionnaire, please make an> address corrections on the 
mailing label found on the copy of the consent form attached to the questionnaire, then return that 
consent form and the quesuonnaire to us at the University in the enclosed, prepaid envelope When 
ihe quesiionnaire /s recened ihc i'nnersiix will send \ou \our Oieck as soon as possible Please keep 
this copv of the consent form for vour own information 



Seattle Social Ovelopment Project 

Parent's Siaiement I understand that participa -on is entirelv voluniarv I have had an opporiunitv to 
ask questions and understand that anv future questions 1 mav have about the research or about sub 
jects rights will be answered hv one of the investigators listed aDove I consent to participate mi this 
aciivitv 



If vou have anv questions about the survev please call Manlvn Hoppc at 543 41 18 



Thank vou. 



J David Hawkins 
Director 



Date 



Please 
print 



f'lrsi name 



I asi name 



Copies to Parent 



Researcher's file 



Parent s Signatuie 



Date 
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intervention begins (Time One), at the cx)mpletion of the program (Time Two), and at some 
designated periods after the intervention is complete (Times Three, Four, etc.). Your evaluation design 
will determine what the critical data collection points will be. For example, if you are using a time- 
series design, you will have more data points than if you are using a pretest-posttest design. 

For some questions, it is possible to gather Time One data even after the program has begun. If 
the information you need is contained in records, you can create your Time One d?Xd ftim these 
records. 

You should collect information on immediate results Just after participants complete the prevention 
program. It is tougher to determine when to collect your long-term outcome data. You will need to 
decide how long after the prevention program to wait if you plan to collect followup in^^^'mation on 
your program's effectiveness. You need to wait long enough to see the effects, but not so long that 
*hey have worn off or have been replaced by other important influences. For example, you might not 
expect to see effects of refusal skills training with sixth grade children on rates of marijuana ex- 
perimentation for several months. Yet by eighth grade it is quite possible that the program's effects 
may have worn off. 

Unfortunately, in the drug prevention research area, very little long-term : ^llowup research has 
been done. As a result, it is hard to recommend the "right" followup period. Frequently, people assess 
drug i5e at the end of 1 year following the prevention intervention. The followup period should be 
determined in your study by the length of time you expect to be required before your program's ef- 
fects would be visible through changes in people's intentions, attitudes, and/or behaviors. You must 
also consider the difficulty of recontacting people at a later date. 

Effort Data 

Remember, effort information tells you about the level and kind of effort going into your preven- 
tion program — the number of hours of parent training you orovided, the names of people who par- 
ticipated in the training, the number of sessions each parent attended, and similar information. Effort 
data let you descril5e what you did to produce the effects you observe in your self-evaluation study. 
They are also needed to answer efficiency questions. You must ask "What do we need to record 
about this program, given our evaluation question?" if you want your program to be something other 
than a black box in your evaluation. 

Effort data collection should occur while the program is being offered. Often, effort monitoring 
becomes part of every working day. Effort measurement must become routine. To ensure daily collec- 
tion of needed information, you will have to design a collection tool that is seen as useful by the 
staff. 

Data Collecting Responsibilny 

Once you have decided what type of data collection instruments to use and how often tc f ollect 
the information you need, you must decide who will be responsible for collecting it. Diffe»eri people 
may be responsible for collecting effort and effectiveness information. If you plan a followup of par- 
ticipants to assess drug use outcomes, you will need to identify a lead person for this activity. Alter- 
natively, prevention service providers may be asked to collect effort or implementation information on 
the services they provide. 

Before you decide who is going to collect the data, you should answer these questions about who 
might collect it: 



54 



■ Hov/ interested are they in the information? If not interested, wh not? 



■ If they're not interested, how will you get the data you ne^d? 

■ How confident can you be in the accuracy of the data collected, given this 
strategy 

The last question is a measure of the level of your staffs commitment to the evaluation. If you 
have identified the data you need, you know who will collect it, and you are confident of their 
ownership of the evaluation and trust that they will provide accurate information, you are well on 
your way to a successful evaluation. Often service rroviders are uninterested in recording things they 
already know on forms! The more you involve your staff in the evaluation process, the better the 
chances are that they will be interested in helping in the data collection. A tool to help you decide on 
a data collection approach for the kinds of data you need is show in figure 1 1. 

Fill out this data collection plan for one piece of effort or process information and one outcome (ef- 
fectiveness) data collection instrument you will use to complete your evaluation. 

Figure 11. — Data Collection Plan 



Instrument 
1. 



What It 
Measures 



Frequency of 
Collection 



Who 
Collects 



Accuracy 
of Data 



2. 



3. 



Part 2: 
Pilot Test 
Instruments and 
Collection Plan 



Once you liave identified your measures, instruments, and collection procedures, build in some time 
to pilot test your data collection system. A pilot test lets you make sure all the instructions and words 
on your forms or questionnaires make sense to people. You can do this by asking either your staf^ 
your family, or your friends to look at them and even to fill out the instruments to see what parts 
are clear or confusing. Once the wording and instructions are clear, try to arrange a sample of 20 or 
30 people similar to people in your project to fill out the instruments. This pilot test will let you 
determine the ability of your data collection instrument to distinguish between people. If all 30 people 
in the pilot test answered a question yes, then the question doesn^t d >tinguish between people and it 
probably should be dropped. A pilot test will help you ensure the reliability and validity of your data 
collection instruments. 

Reliability is your assurance that your instrument or measure is consistent. Beginning an evaluation 
with an unreliable instrument is similar to a carpenters building a house with a rubber ruler. If you 
are testing a questionnaire, and you give it to the same person twice within a short timespan, you 
would expect that person to give very similar answers on the questionnaire. If this doesn't happen, 
you probably have an unreliable questionnaire. This is called test-retest reliability. 
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Part 3: 

Monitor Field 
Reports on 
Measures and 
Procedures 



Another kind of consistency or reliability is the internal consistency of items all seeking to measure 
the same concept. If you are seeking to increase family bonding and you have several questions on a 
survey to measure bonding, you would expect these questions to be generally answered the same way 
(i.e., consistently) by individuals. This is called internal consistency and is measured by a statistic call- 
ed Cronbach's alpha. 

Just because an instrument or measure is reliable does not ensure that it is valid. Validity is your 
assurance that an instrument measures what it is designed to measure. If you administer a self-report 
drug use questionnaire to known drug abusers, and discover that less than 5 percent of your sample 
reports any drug use in the past year, you should seriously question the validity of your instrument. 
Conversely, if you administer a drug use questionnaire to first graders and discover that 74 percent of 
your first graders have experimented with drugs or alcohol, you probably have an invalid measure. 

Establishing the reliability and validity of new data collection instruments is time-consuming. That 
is why, whenever possible, you should use existing instruments in your self-evaluation study. 

A pilot test allows you to answer questions beyond reliability and validity of measures. It is a 
systems check that encourages your staff/team to return and say "this won't work, because . . /' Tak- 
ing the time to troubleshoot the process of data collection before your evaluation begins enables you 
to debug the evaluation in advance. Each time a member of your team suggests an improvement to 
the process, his or her individual ownership in the process wHl increase. 

Reliability and ease of collection need to be assessed through field tests of instruments and pro- 
cedures, with adjustments following if needed. 

Staff/team agendas 5 and 6 are included at the end of the chapter to help you collaboratively plan 
for quality data collection instruments. 

Describe your plan for pilot testing >our measurement instruments and ensuring that youi data col- 
lection system is workable here. 



Instrument Pilot Test Pl?.n 



What are the instruiTients we will be using? 



2 



Who will read them to make sure they are appropriate for our target audience(s)? 



3 



Where will we find a sample of 20-30 people to fill out the instruments? 



4 



How will we know if the instruments are reliable? 



5, 



How will we know if the instruments are valid? 



56 



fi4 




1 



STAFF/TEAM AGENDA 5 
ORGANIZING AND COLLECTING DATA 



ATTENDING: Fl.. a small progiam, the whole team or staff. For a larger program, representatives 
from each unit. 

TIME: Schedule 2 hours. 

OBJECTIVES: To ensure that all people know what information n^ust be collected and why this in- 
formation is critical to the success of the evaluation. To create a plan that will result in the efficient 
collection of data over the course of the evaluation. 

I. Review with your whole team the evaluation plan and how each of the data collection measures 
relates to the success of this plan. Service providers will be asked by their clients, "Why is this 
necessary?" If your program staff know the answer, clients will be much more willing to respond 
openly and honestly. Also, measures of staff effort — "What did I do today?" ~ may be threatening 
to staff members who don't understand why the information is collected. 

II. Specify who will need to collect what, and pass out a schedule of collection times. Be sure to allow 
time for discussion from the staff members who will be providing the information. Your goal is to 
have everyone in the information chain become owners of the process. The greatest threat to the in- 
tegrity of your evaluation is insufficient or incomplete information. Without consistent data collection, 
you will not be able to relate your outcome results to program activities. 

III. Design a plan to monitor data collection. Members of your team or staff must feel that giving the 
information they are asked to provide is an important part of their jobs. A plan that systematically 
follows up each week or every other week on the submission of data will reinforce the importance of 
data collection. (A little time given each week for team members to share with others "what they've 
done this week" will be a public recognition for both effort and measurement of that effort.) 

IV. Close the meeting with a plan for a 1-week or 2 week pilot test of the instruments to be used and 
the data collection system. You will find that the information from service providers in the field will 
improve the validity and the efficienc> of data collection If team members' suggestions are i-iCor- 
porated into the design of instruments or into the systems of collection, they are much rnor^ likely to 
become "owners" of the evaluation process. 



STAFF/TEAM AGENDA 6 
TROUBLESHOOTING MEASURES AND COLLECTION PROCEDURES 



ATTENDING: Team members who have worked in the design of the evaluation and others who 
have specific input from the pilot test of the measures and collection process. 

TIME: Schedule 1 hour. 

OBJECT iVE: To review the experience of the pretest period and adjust measures and collection pro- 
cedures to ensure an efficient ongoing evaluation. 

I. Review the feedback from staff/team members on the pilot test of evaluation forms. Listen to your 
staff/team's input on procedural improvements of the data collection process. Adjust the instruments 
and procedures based on this input. 

II. Draft a memo to all staff/team members outlining the changes in instruments and data collection 
procedures. 
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Chapter 8 



Data Analysis 

Step 5: Analyzing the Data 

Before you ever begin to collect data, you should have an analysis i^lan. If you wait until after you 
have collected your data to think through how you will analyze it, you may find that the analysis 
you want to do demands information that you failed to collect. Conversely, you may find that you 
have taxed your staff/team unnecessarily in collecting data that are of no use to your analysis. 

Data analysis should be a rewarding activity. This is when you and your group will know if, in 
fact, you've made a difference in preventing drug abuse. Designing the analysis plan will maximize 
the chance of finding what difference your program has actually made m the lives of the people you 
serve. If your data analysis is incomplete or inadequate, you may fail to see the true effects of your 
program. 

^^^^ ^ Your data analysis plan is your plot for summarizing the information you collect to answer the 

Choose Analysis questions you ve asked There are two kinds of information that you may choose -o report: descrip- 

Procedures tive statistics and relationship statistics. 

Descriptive Analysis 

Descriptive statistics answer the questions. How much? How many? What was the average? What 
was the rate? In what proportion? How much variation was there? 

For example, descriptive statistics are used to answer these questions: 

■ "How many children reported knowing someone who uses marijuana?" 

■ "What proportion (or percentage) oi our high school seniors reported drinking 
alcohol daily?" 

■ "What is the mean score on the Family Cohesion score of the FACES instrument 
for families trained in our program?" 

The first task in preparing descriptive statistics is basic addition, add up the nutnbers in categories 
of interest and report the frequencies. Examples include the following: 

■ number of subjects in our study 

■ number of subjects by age, sex, or ethnic group 
B number of preventive services \vc provided 

■ number of lessons taugh' 

■ number of parents in our parent classes 

■ number of students who took a dry pledge 

■ number of students reporting problems with alcohol or other drugs 

You may decide to report the numbers in terms of proportions or rates in each category. A per- 
centage is one kind of rate. A rate is simply a fraction or .atio. It represents the number of times an 
event occurs (use of drugs, being suspended, etc.) divided by the total population of interest. 
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Parent Participation in Prevention Program by Ethnic Group 

Ethnicity Pe/centage of Parents 

Black ?A 

American Indian 7 

Asian 15 

Hispanic 12 

White 42 

A third way to display frequency data is through a visual display such as a graph. If you make a 
graph, be sure that it is labeled clearly. A graph has the advantage of allowing the reader an im- 
mediate comparison of the frequencies reported that might be less apparent in a table or list of 
numbers. Figure 12 displays three graphs showing proportions of a sample with different drug 
outcomes. 

Figure 12,— Proportion of Population Using Drugs: Three Sample Graphs 



Bar Graph Pie Graph Line Graph 




Incidenis of Use Drug Free Occasional Frequent 

Incidents of Use 



Measures of Central Tendency 

To summarize frequencies, a measure of central tendency is used. Three measures of central 
tendency are the mean,, mode, and median. 

The mean is the most commonly reported measure of central tendency and is the simple arithmetic 
average of a list of values. Add the list of values and divide by the number of entries to get the 
mean. 

Since the mean, or average, is subject to being affected by extreme high or low scores, the median 
or middle score is at times reported. If you listed 15 scores from low to high, the median would be 
the eighth score listed. 

The third measure of central tendency is the node or the score that occurred most frequently. The 
mode IS rarely reported m evaluation studies. 

For example, if you have the followmg scores on a five-poini scale from 10 people: 
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Person 




Score 


#1 




1 


#2 




3 


#3 




3 


#4 




4 


#5 




1 


#6 




5 


fl 




4 


#8 




1 


#9 




3 






1 




Sum 


26 



26 

The mean is — = 2.6. 
10 



The median is 3. 



The mode is 1. 
(There are four cases with a score of 1. 



Measures of Dispersion 

Another important descriptive statistic is the standard deviation. The stundaiu deviation is a 
measure of the dispersion or variability of scores. It gives ^ou an idea of the distribution of scores 
without requiring that you list all scores. 

The formula for calculating the standard deviation is: 



N-1 

The utility of presenting a measure of dispersion can be illustrated through the example of an 
evaluation that compared client satisfaction between two counselors. Each counselor was rated by 
clients, on a five-point scale, with a "5'' being "satisfactory in all ways." The standard deviation is 
calculated by first adding up the raw scores (19 for both counselors) and dividing by the number of 
scores to attain a mean (M = 3 Next, each individual's score is subtracted from the mean score to 
attain an X-M score. The X-M score is squared for all clients, and these squared scores are added up 
to form the sum (Z) of (X-M). The sum of (X-M) is entered into the standard deviation formuh, 
where it is divided by the number of scores minus 1 (in our example, 5-1=4). The square root of 
this number is the standard deviation. 

CMent Satisfaction 



Counselor #1; 


Raw Score 


(X-M) 


(X-Mt2 


Client 


3 


-.8 


.64 


Satisfaction 


4 


.2 


.04 


Scores 


4 


2 


.04 




3 


-.8 


.64 




5 


1.2 


1.44 



I=i9 
M = 3.8 



1 = 2.8 




= .83 
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Counselor #2: Raw Score (X-M) (X-Mr 

Client 2 -1.8 3.24 y 

Satisfaction 5 1.2 1.44 /lOS 

Scores 5 i.2 1.44 SD="\ /— = ^64 

2 -1.8 3.24 V 

5 1.2 1.44 

1=19 1=10.80 
M = 3.8 



In this example, even though both counselors have the same mean satisfaction score (3.8), the stan- 
dard deviations of scores (.83 versus 1.64) suggest that one counselor is getting consistent responses 
from her clients, while the other counselor's ratings vary widely. This information may be useful in 
supervision of the counselors. Why is one counselor getting such extreme scores? Does he work better 
with certain types of individuals? What can be done to decrease the variability in clients' responses to 
this counselor? 

A second measure of dispersion is the range. The range is simply the distance between the highest 
and the lowest score. Since it reflects only two scores, it is not as valuable a statistic as the standard 
deviation, which conside»'s all scores. 

If your evaluation asks only effort questions, you may need to report only means and standard 
deviations to answer the questions youVe raised. If you are concerned, about your program's effec- 
tiveness, however, you will probably need to employ a relationship statistic to answer the question 
"Did we make a difference as compared with other programs, past performance, or a predetermined 
standard?" 

Examining Relationships 

In examinmg frequencies, you looked ai individual variables. In examining relationships, you ex- 
amine possible relationships between things. Tnrough examining relationships, yo; 'n discover 
whether attending parenting classes reduces family drug use or how the teaching c ^^roblem-solving 
skills in classrooms affects the suspension rates of middle school students. 

To know if there is a relationship between your prevention program and the outcomes you seek, 
you will want lo answer three relationship questions: 

■ is there a difference following our program between tne groups we served and 
those with whom they are compared? 

■ Is that difference meaningful in a practical sense? 

■ Is that difference statistically significant? 

In the following section, three methods of summaiizing relationship data are discussed: contingency 
tables, comparisons of means, and correlational approaches. Also in this section, four basic tests for 
statistical significance are discussed. You do not need to learn he to do these statistical tests. You 
probably will want to learn the purpose for which each test is useu, however, so that you can make 
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informed decisions about whether and when to ask for assistance from a resource person in com- 
pleting your analyses. 

Contingency Tables 

One way to investigate the relationship between variables thai interest you is to build a contingen- 
cy table. A contingency table presents the number and proportion of people falling into two or more 
categories on two or more nominal variables. Nominal variables place people in categories, such as 
male versus female, or people receiving services versus people not receiving services, or people who 
have used drugs veresus people who have not. For example, if we were interested in the relationship 
between the teaching of problem-solving jkills to seventh grade students and their rate of suspension 
from school, our table might look like i' figure 13. 



Figure 13.— Effect of Teaching Problem Solving Skills on Suspension Rates of Seventh Grade 
Students (1985 86): A Contingency Table 





Taught 


No 






Problem Solving 


Problem Solving 


Totals 




N = 268 


N = 276 




Number and proportion of 


14 


36 


50 


students suspended 


(5.2%) 


(13%) 


(10%) 


Number and proportion of 


254 


240 


494 


students not suspended 


(94.8%) 


(87%) 


(90%) 




268 


276 


544 


Totals 


(100%) 


(100%) 


(100%) 



In this table, both actual counts of students and proportions have been listed. The contingency 
table shows that only 5.2 percent of the students who were taught problem-solving skills were 
suspended during the evaluation year, while 13 percent of the students who did not receive training 
were suspended in the same year. Overall, 10 percent of the student body was suspended from school 
in the 1985 86 school year. 

Contingency tables can be developed to answer a number of different questions for a number of 
different variables. If you wanted to compare long-term and short-term suspension, for example, you 
would simply need to add another row to the table. 

Is the difference shown .n figure 13 meaningful in a practical sense? In this case, cutting the 
suspension rate of one group of students from 13 percent to 5.2 percent would certainly appear to 
have practical value. Prevention v/orkers who found these results would probably want to recommend 
that this problem-solving curriculum be expanded to include all seventh grade students. 

If you do identify a difference that appears to be meaningful, you may want to go one more step 
and determine if the differences are statistically significant. Tests of statistical significance tell you 
how likely your results were to have been obtained by chance alone. Traditionally, findings that have 
a greater than 5 percent likelihood of happening by chance alone are considered nonsignificant. 

Do not confuse "meaning'' or "practical importance" with "statistical significance." If the dif- 
ferences in your contingency table are not large enough to be of practical value or importance, little 
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reason exists to move to the next step of testing for statistical significance. In such a situation, testing 
to see if the results occurred by chance alone is unnecessary. 

When you display your data through a contingency table, you usually use a chi-square test to 
determine statistical significance. In the contingency table displayed in figure 13, the chi-square value 
was statistically significant beyond the 5% level. 

Statistical Significance 

How important is it that your program is shown to make a difference that is statistically signifi- 
cant? If you operate under a grant that expects statistical tests of significance, or if you expect to 
publish reports of your program in professional journals, demonstration of statistical significance is 
important. For many purposes, your knowledge that the differences are meaningful in a practical 
sense is a valuable outcome in itself. 

Statistical significance is often a tough test to pass. It requires either large differences between 
groups or a large number of participants in your study. Both of these factors work against small 
prevention programs finding statistical significance in self-evaluations. 

Comparing Means 

A second kind of comparison you can make between groups is a comparison between means. For 
example, you may want to compare the average frequency of alcohol use between students who 
received a prevention curriculum and students in classes where the curriculum was not offered. 
Assume that at the conclusion of the 1 0-week curriculum you asked each student to report the 
number of times he or she drank alcohol in the past month. By adding all the scores for each group 
and dividing by the number of students in that group, you would get the mean frequency of alcohol 
use for each group. You might find that students exposed to the curriculum reported an average or 
mean of three drinking events during the month, while the mean frequency of drinking in the com- 
parison group was five times. Again, once you find such a difference, you need to decide whether this 
difference is large enoagh to be meaningful or important. You may also want to see if the difference 
is statistically significant. 

When comparing means of two groups lur statistical significance, you use a statistic called the t- 
test. If you are comparing more than two groups, you would employ analysis of variance, and if you 
wanted to compare more than two groups while controlling for some baseline measure, you would 
use a statistical procedure called analysis of covariance. 

Read the following three examples, and decide which statistic you would use to test for 
significance. 

Example 1: You are the developer of an innovative program to reduce adolescent drug use by 
building stronger hr,me attachments through the teaching of communication skills to parents. You 
work with three groups of parents: a group in the Hispanic community, a group in the Black com- 
munity, and a group in the Italian community. Although the three groups all have children of the 
same age, they vary greatly on measures of income and education. In the class, you administer a 
pretest and a posttest, that measures preferred problem-solving strategies when theyVe faced with a 
conflict with their adolescent child. You also ask them to tally the number of conflicts they have 
with their children for a 6-week period after the end of your class. You want to compare the three 
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group means on the posttest, as weii as the group means for the number of conflicts talhed. Since you 
think that variables such as education may be a powerful factor in the problem-solving skill of 
parents, you decide to control for those variables in each group. The test of significance >ou would 
use is . . 

Example 2: You coordinate a prevention program that trains teen peers to counsel classmates about 
the risks of drug and alcohol use. A school district with two high schools has asked if you would test 
your program in one of their high schools. You decide to work only with the sophomore class in 
your first year. You hope to change both rates of student use of alcohol and drugs and student at- 
titudes about substance abuse. The district has agreed to let you use the sophomore class from a se- 
cond high school as your control group. At the end of the school year, you administer a survey to 
sophomore students in both the project school and in the control school. You want to compare the 
means of the two groups on each question to see if your program has made any significant dif- 
ferences in either drug and alcohol use or in attitudes about drug and alcohol use. The test of 
significance you would use is 

Example 3: You coordinate county-wide substance abuse prevention efforts. Within your county are 
ihree programs that all compete for a very small pot of funds that you have to distribute for county 
drug and ala ' ol prevention efforts. The programs are: a police department drug education program 
featuring a presentation by a former undercover narcotics agent, a school-focused assertiveness train- 
ing program that uses role play to teach young people how to say no, and a parent group that spon- 
sors athletes and other role models to come and talk in classrooms about healthy lifestyles, '^he two 
things that are common about all three programs is that they happen in schools and target middle- 
school-aged children. How do yiou decide who gets the funds? You decide to design an evaluation 
that will test the impact of each of the programs. You work with the principal and discover that a 
near random assignment of students has taken place in the scheduling of eighth grade social studies 
classes: You divide the eighth grade social studies classes into four groups; one will e ' the police pro- 
gram, a second will hear roli models, a third will learn how to say no, and the fo group will be a 
control group. You administer student drug use attitude surveys as posttests to ea 3up. You want 
to compare the means on the posttests for each group to determine which group .le greatest im- 
pact and is most deserving of your funds. You will use the 

statistic. 

Answers: ( 1 ) analysis of covariance, (2) t-test, (3) analysis of variance 

You may choose to use a correlation or regression coefficient to test the strength of a relationship 
cetween two ordinal or interval level variables. These are variables that measure amounts, frequen- 
cies, or scores on scales with increasing values. Regression approaches tell you whether one variable 
increa^'*" when you systematically increase or decrease the amount of a second variable. For example, 
if you increase the number of alcohol prevention lessons taught in the ninth grade, is there a cor- 
responding decrease in the number of students referred to the office for alcohol-related offenses? Cor- 
relation simply tells the extent to which a straight line or linear relationship exists between two 
variables. For example, is there a relationship between the frequency of marijuana use and the fre 
quency of alcohol use in your sample? 

The strength of the regression coefficient or correlation between two variables can range from a 
+ 1 to a - 1. If for each jditional lesson you taught, there was a corresponding 10-percent reduction 
in alcohol-related referrals, you would have a perfect negative correlation of - 1. If each additional 
lesson you taught had a corresponding 1 0-percent increase in referrals, your regression coefficient 
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would be + 1. If the regression coefficient is 0, you know that no relationship exists between the 
number of alcohol prevention lessons and the number of alcohol offenses in the school. 

If you are examining regression roefficienis or correlations to assess your program's effects, it is im 
portant to remember that all you can say is that a relationship exists between the two variables. 
These statistics don't tell you whether the program or some othe'' factor caused the relationship you 
observed. 

To say that your program caused the outcomes you observed, you must be able to show three 
things: (1) a strong relationship exists between your program and the outcome you measure, (2) your 
program must have occurred before the change took place, and (3) you must rule out alternative ex- 
planations for the observed relationship. 

Now, summarize your data analysis plans on the following summary sheei. 

Data Analysis Summary 



1. Write *** THE QUESTION here. 



2. Who's interested in my results? 



3. What descriptive information will I report? 



2. 
3. 
4. 
5. 



4. What m'-.thods for analyzing relationships will 1 use? 

a. Contingency tables? 

b. Comparisons of Means 

c. Correlations? 



5. How will 1 define a difference that is meaningful in a practical sense? 



6. Which test of significance will 1 use? 



Why? 



Part 2: 

Design Tables 
and Graphs for 
Reporting Results 

Part 3: 

Begin Collection 
of Data 

Part 4: 

Monitor Data 

Collection 

Regularly 



Part 5: 
Analyze Data 
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After you know what analysis approach you will be using with your data, and before you ever 
begin to collect the data, you should know how (what formats) you ar'^ going to use to report your 
findings. In chapter 9, you will have the opportunity to fill in these preliminary graphs and/or tables. 

As you can see, most of the work of an evaluation is done iong before you collect your first data. 
Doing the hard work first ensures that you won't expend energy collecting data you don't need 
Knowing why you need the data and how you will use it also gives meaning and purpose to the pro- 
cess for your staff/team. 

Don't assume that the inhrmation you need is being collected as planned. If your evaluation covers 
a program that operf.tes for oi:ly a few weeks, your staff/team will probably maintain a high degree 
ot commitnient to the data collection plan. However, if your evaluation period is longer than a 
month, you will need to review regularly both the purp</se of the data collection and the procedures 
People have many demands on their time, and your staff/team's motivation for collecting information 
may wane if it isn't regularly reinforced. 

A good way to keep your staff/team's interest in the evaluation process - to plan feedback sessions 
of preliminary findings as soon as there's something to report. (You car li rt such descriptive 
statistics as "the number of lessons taught this week" or "the number of parents who have par- 
ticipated in our program during the first 3 months ") Using moni'oring data to reinforce the eifort 
that your staff or team is making will help them sec that these extra duties are a priority. You should 
keep your evaluation chart on display so everyone can identify where they are in the process. 

Once your data are collected, you will need to ensure that you aren't missing some data before 
turning the information over to coders or data processing specialists. Some of the data may need to 
be transferred to forms that can be easily keypunched or entered directly into a computer. If you are 
planning hand tabulation, you might consider an afternoon work party involving many of those who 
were assigned to collect the data. Delegate tasks! The more your staff participate in this evaluation 
the more they are going to take ownership in the process. 

If your study is not too large, you will ge'. some initial findings almost immediately. Calculating fre- 
quencies and filling in contingency tables will give you an indication of what you've accomplished 
before you attempt a statistical test. 

The actual data analysis should be routine, since you have already planned each step of this pro- 
cess before you began collecting data. Staff/team agendas 7 and 8 will help you in planning and com- 
pleting data analysis. Both agendas follow the end of this chapter. 

Computers and Consultants 

If you use a packaged computer analysis program, such as the Statistical Package for the Social 
Sciences (SPSS), you can obtain a significance test as well as descriptive statistics, such as frequencies, 
measures of central tendency, standard deviations, and correlations between variables your study 
investigates. 

This handbook does not attempt to teach you how to do regression analysis, analysis of variance 
or any other statistical test. What you need to be able to do is recognize which analysis routine is ap- 
P'opriate. The analysis can then be done on a computer by a person familiar with basic statistical 
?'-2lysis packages. This is a good time, however, to consider contacting one of the resources men- 
tioned for technical assistance: expertise on your staff or local colleges and universities. 
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You will not need access to a mainframe computer for mauy of the tasks you have before you. A 
personal computer with a hard disk has the power to analyze data for the vast majority of prevention 
programs. If you have access to larger systems, your data can probably be processed faster; however, 
your processing costs may be higher. 

Factors to consider in deciding what computer resources you will need include the following: 



■ Number of subjects 

■ Amount of data per subject 

■ Number of statistical analyses 

■ Frequency of reporting 

■ Availability of . . . 

— Funds 

— Personnel 

— Equipment 

— Analysis software programs such as the Statistical Package for the Social 
Sciences (SPSS) 



If the number of subjects, amount of information, and number of statistical analyses lo be done are 
large, use a computer for analyses. For smaller projects, a hand calculator will do the job. 

HELP!}— A Cry from the Unnumbered Wilderness 

At this point, you may think, "Is statistical analysis really necessary?'' The question of whether it is 
really necessary should be answered by your purpose/audience matrix. When you recall who your au- 
dience is and the purpose for which the information is needed, you may find that statistical analyses 
aren't necessary. Remember, your question and your purpose for doing the evaluation will determine 
how much statistical analysis is needed. 

A very pragmatic reason exists, however, lor engaging in statistical testing in evaluations of preven- 
tion programs. Unlike treatment programs, prevention programs cannot point to case studies of 
recovering clients, rates of reduced recidivism, or numbers of clients employed to show that preven- 
tion has made a difference. Ultimately, prevention workers have to demonstrate that rates of drug ex- 
perimentation, driving under the influence of alcohol, or other drug-related outcomes in target popula- 
tions have changed. Statistical tests aiiow you to say whether the results you found were likely to 
have occurred by chance alone or were more likely the result of other factors, such as you; preven- 
tion program. 
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STAFF/TEAM AGEND' 7 
PLANNING FOR DATA ANALYSIS 



ATTENDING: Staff members who have been a part of your evaluation team. Other staffers with an 
mterest in data analysis or individuals with a social science research background who may serve as 
resource people for you. (All participants shculd have read chapter 8 of the .'ndbook before this 
meeting.) uvi^iv. una 

TIME: 2 hours 

OBJECTIVES: Given your evaluation goals and objectives, a model of your intended outcomes and 
instruments to measure effect and effort, create a data analysis plan that will tell you whether your 
program has made a difference. 

I. Consider the resources thai are available to you for data analysis-personal computer mainframe 
computer, computer time, chapter 8 of this handbook, local expertise. 

have^wlleded '''^^'^ Procedures that will answer the questions that you have asked, given the data you 
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STAFF/TEAM AGENDA 8 
ANALYSIS OF DATA 



ATTENDING: Staff members who have been ? part of your evaluation team. Other staffers with an 
interest in data analysis or individual with a social science research background who may serve as 
resource people for you. (The various data sets that are going to be used in the analysis should be 
brought to the meeting for review of completeness.) 

TIME: 2 hours 

OBJECTIVES: To ensure that all data are collected and in a form acceptable for data analysis. To 
delegate responsibilities for completion of the analysis and establish a time line for its completion. 

I. Review the d?:<5 collected over the evaluation period to ensure that you have complete data. If in- 
formation is miising, establish responsibility to recover missing data, or document why some data 
might not be present. 

II. Delegate the analysis tasks. Establish a time line for completion of the analysis. 



70 



Part 1: 

Complete "Dummy' 
Graphs and Tables 
for Reporting 
Findings and 
Plan Presentation 
of Data 



Making Use of What You've Learned 

Step 6: Reporting the Findings 

In Step 1, you filled in a purpose/audience matrix that specified who was interested in your results 
Now IS the time to use that matrix to design your presentation 'o address your evaluation audience 
Reports designed for your boss, or your board of directors, or your line staff, or your local media 
representatives all require customizing the presentation. 

Don't underplay the importance of what you've found. These results represent a considerable in- 
vestment by everyone in your program. If one of your audiences is your staff/t^am, you may want to 
schedule a I -day retreat to present the findings in the morning and structure problem-solving sessions 
in the afternoon to create a plan for applying what you've learned. This is also an opportunity to 
debrief the evaluation process. Staff/team participation is as important at the end of the process as it 
r?/^ the beginning of the process. If you are a program manager, you don't have time to be in the 
leld each day to ironitor your evaluation. Because of that, some of the results you find will be hard 
^^*^^the input of an staff who were involved, however, the meaning and the 
implications of the findings will become more clear. 

When you report findings to those who participate in the program, you can assume a level of 
knowledge that your board of directors or the local media may not have. In all cases, you should 
begin your report by "renewing the mission" or pointing out why you asked your particular evalua- 
tion question and what use you expect to make of the answer. You are then able to place this 
evaluation in the larger context of the overall program goals. If you report the findings of the evalua- 
non without explaining how the answers you've found can be used by your target audience, you may 
be inviting a response of "So what?" Be sure to tell your audience why you asked the question, what 
you learned, and what you expect the audience to do with this information. 

T^ZT"^^'' provide both an oral presentation of the results and a written summary as backup. 
Too many evaluation studies end up as shelf or file fillers-unused, in part, because people do not 
have or take the time to read them. ^ r , ^ pcu^.c uo noi 

r.^^ll^^'^^l^'f'' oral information to written information. Whether they are legislators, program 
managers, or board members, people Iikc to be told what you've found and what you see as the im- 

r!S Zm II?""- P'°P't^^"' '° "^""^^ "^'^^^ '^^ "^hat do you want us to do?" 

Uive them both answers. Keep the oral presentation relatively simple. Share the highlights of your 
results, and describe what they imply for program change, maintenance, funding, etc., given your pur- 
pose and audience. Then, distribute a summary sheet of results. If you have invited the press to hear 

most im^Sam" ''""""^ "^'^"^^'"^ ^ '^^^^'^ ^^^^ ""'^'"^ 

foAU^l'^'"'' 'i""" ? yo"-" P'-esentation of the evaluation results. Target this presentation 
to. the most important audience you identified on you. purpose/audience matrix. 
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REPORTING YOUR RESULTS 
1 . Write down your evaluation question here. 



2. Draw one graph or table to display the results you hope you will find. (Consult the model 
graphs i?i chapter 8 for examples.) Be optimistic here. It's your chance to have the data come 
out just as you want. 



3. Using the visual displays you have created, plan a presentation for your audience. In the presen 
tation, tell them: 

a. What you set out to learn— your question. 

b. What you actually found. 

c. What you think these results mean to this audience. 



Presentation Outline 
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Part 2: 

Role Play Presentation 
to Target Audience, 
Adding Data Needed 
to Respond to 
Audience Questions 



Part 3: 

Report Your Findings 
to Identified 
Audiences 

Part 4: 

Adjust Your Program 
Based on What 
You've Learned 



Now, practice your preservation with your ^taff/team. Have your team members take the roles of 
your target audience, and role play your presentation to them. By doing this now, you will identify, 
before you even begin your evaluation project, the kinds of questions and reservations your target au- 
dience is likely to raise. Having anticipated these questions through this role play se^^^ion, you can 
decide whether additional information should be collected during the evaluation to answer such ques- 
tions. In short, by preparing now to use the evaluation results for the very purpose for which your 
study was designed, you will g,uarantee that the informatioii you gather can be successfully used for 
this purpose. 

You should role play one additional scenario. Change the information in your example graph or 
table so that it comes out just the opposite of what you hope you1l find. Now what will you do? Fre- 
quently, the first reaction is: "Bury it! Let s hide the information, and forget that an evaluation was 
done.'' But, if you look at that information again, you are likely to find that it is very useful after 
all— perhaps not for the purpose for which you originally de^^igned it, but for another, equally impor- 
tant use. For example, information that you gathered in hopes of showing your program's success in 
preventing drug experimentation among seventh graders, in order to guarantee ^ iture funding for the 
program, may show that students in your program were as likely to experiment with drugs as 
students you did not serve. While the information may not be useful to obtain funding, it will very 
likely stimulate rethinking regarding the program itself and how it should be altered or improved to 
better achieve its goal of reducing drug experimentation. 

Unanticipated findings need not be detiimental to the self-evaluating organization. They give you 
information to improve your work. Showing others that you seriously :eek to achieve your goals and 
are willing to gather and use information on program outcomes to improve vour prevention services 
demonstrates a commitment to self-improvement. Learning from the negative outcomes will build 
your credibility as an organization seriously committed to the goal of drug abu^e prevention. 

You should now be prepared to confidently discuss both the findings and t!ie implications of your 
program. Projecting vour tables an'i/or graphs on an overheati projector an excellent method of 
making the results and implications of your outcomes clear for your audience. 

Congratulations! YouVe just completed your first evaluation plan. The final step is to take what 
you've learned from this evaluation and apply it to the improvemcni of next year's program. That 
process may begin with an all-staff meeting, or it may involve selected representatives froir. your 
team. 

Staff/team agendas 9 and 10 are included at the end of this chapter to assist your planning for the 
presentation of results. 

Since your first evaluation experience has been successful, you may be tempted to greatly expand 
the scope of your next evaluation. Reflect for a moment on the joys of keeping n simple, however. 
One of the reasons for your success in your first evaluation is that you kept it simple. Beginning 
again with one or two new questions to answer and using any additional energies to ensure that the 
improvements suggested by the first evaluation become a reality would comprise a sensible plan. 
Nothing can be more discouraging to a staff than to give energy to identifying needs of program im- 
provement and not to see those changes implemented. 

This handbook will be a resource guide for future evaluations. What you have accomplished with 
one hypothetical question is only the beginning. 

Enjoy evaluation! 
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STAFF/ TEAM AGENDA 9 
REPORTING THE FINDINGS 



ATTENDING: Your whole staff or team and those individuals who have assisted you in the evalua- 
tion effort. This is one of the payoffs for the people in your group who have given extra time to en- 
sure that you have complete, high-quality data. This should be a major event, perhaps coupled with a 
staff dinner or potluck lunch and time to taliv about what it all means. 

TIME: 2 to 3 hours 

OBJECTIVE: To share the results of your evaluation. 

Before your meeting, review all of your results. Do not bury outcomes that seem to reflect negatively 
on your program. They may be the key to makmg important improvements in your program. Also, 
don't overlook unintended outcomes. You may find some significant side effects of your program that 
were not anticipated by the questions you asked. After reviewing the data, have some preliminary in- 
terpretation to share with your team. 

Decide who else might be interested in the findings of your evaluation. In a second meeting, you may 
want to include community representatives who may have an interest in your program, potential 
funders, and news media representatives. Don't forget to invite the participants or clients of your pro- 
gram as appropriate. They, too, have a large investment in your success 

Once you know to "whom" you will be presenting your data, decide on the formats that will besi 
display the effects of your program— charts, graphs, text, overhead transparencies. 

I. Review with your staff the model of the program and how this evaluation speaks to certain 
elements of the model. 

II. Discuss the implementation results first. "This is what we did this year." That in itself is a sum- 
mative report of effort that will give your team a greater appreciation for what can be accomplished 
by doing your job week in and week out. Too often, people get so caught up with the small problems 
of weekly work that it is hard to appreciate what has been accomplished over the long haul. 

III. Discuss the outcomes, the effects of your group's work. This is one time in the year that you can 
say with some confidence that you have made a difference! Share the negative as well as the positive. 
Remember, evaluation is a way you improve what you do. Your service providers will be a great 
source of information as to "why'' negc tive effects, noneffects, or unintended effects were found in an 
evaluation. 

IV. Interpret the data from your persp ;ctive, and then invite your team to contribute their interpreta- 
tion' from the numbers that have been presented. Together, you will build a clear picture of what 
your program accomplished and how i. was done. 

V. Solicit input from all your team on how the program and/or the evaluation process may be im- 
proved from this first experience. 

VI. Celebrate the accomplishments of your program and youi staff! 
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STAFF/TEAM AGENDA 10 
MAKING USE OF WHAT YOU'VE LEARNED 

ATTENDING: All staff members involved in planning programs. 

TIME: Schedule 3 hours. 

OBJECTIVES: To use the results of this year's evaluation of effort, effectiveness, or efficiency ques- 
tions in revising programs and procedures for the following year. Evaluation information should serve 
as one of the determine, r*" program revisions. (Other information to be considered at this planning 
me ing includes: changing needs of client population, program mandates froui funding agency, recent 
lovations in service models, budget projections.) 

I Review the theoretica! model of your program. You always need to be prepared to tie the 'what" 
you plan to do to a mcdel that explains "why" it is important that this be done. This can be par- 
ticularly ortant in lookinc at evaluation results that usually speak to only part of a prograiP. Once 
you "see" that you have iswered r.oim questions about one part of your progr.ai, it may be easier 
to identify whi<-h part of the prograni .cserves a closer look in the following year. 

IL After you ha considered the effects uf your evaluation, weight the effort and the efficiency of 
achieving those cjtcomes, to determine if your agency's resources were well spent. 

Make the changes that the evaluation findings suggest. 
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Evaluation Think Sheet 



1. Close your eyes and try to picture your image of evaluation 

Now, take a minute to label that image and write it in the space below: 



2. Before we begin, do you have questions that you expect this manual to ansiver for you? List these 
questions below: 

1. 
2. 
3. 

3. At this point, draw a simple model of your prevention program following the example displayed in 
figure 2, chapter 3. 

A Mode! of My Prevention Program 
Resources Program of Services Immediate Results Outcomes 
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4. 


Think about the nagging questions you have personally about your program or agency. Wrue two 
or three of them down in the space below: 

I Wish I Knew 




1. 

3. 


5. 


Which of these questions are questions of effort? Which are questions of effectiveness? Which are 
questions of efficiency? 


6. 


Jot down two questions you have about your program's efforts: 




1. 
2. 


7. 


Jot down two questions you have about your program's effectiveness: 




1. 

2. 


8. Jot down two questions yon have about your program's efi'iciency: 




1. 

2. 




9. Complete the purpose/audience matrix. 



Purpose/Audience Matrix 

Before deciding on an effectiveness question, identify why the information is needed. Why do we need to 
answer this question? What could we do with the information? This matrix attempts to help you clarify 
answers to these questions by outlining possible uses to which iniormation may be put in an agency and 
possible audiences lo whom the information might be addressed. 

Place an "x" in each box of the matrix that represents the intersection of a purpose and audience for which 
you believe the answer to your question will be useful. 

Purpose 

Management 
Control and 

Planning & Operational Secure Public 
Development Decisions Funding Relations ??? 



Management 



Service 
Providers 



^ Funders 
u 

d 

' Other 
€ Agencies 
n 

^ Public or 
Community 



Participants 
or Clients 
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10. What is the effectiveness question youVe chosen for your evaluation? 

***The Question*** 



ERLC 
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1. Complete the following exercises to choose your design and define your sample: 



A. Considering the designs we've discussed, .vhich one do .--^ think will best answer the 
question you're asking? 



B. Diagram that d.3ign in the space below, labeling the assignment process, the pretest, post- 
test, and treatment. 



C. How will you draw your sample for your evaluation? 

1. To what population should your answer apply? 

2. What sample size will you need? 

3. Ho7' will you draw your sample? 
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12. Operational ize your qucslion bO vou will know whal specific behaviors >ouVe going lo measure as 
outcomes of your evaluation. 



! 

Write "THE QUESTION" here: 



List the long term outcome variables in your List specific measures of the variables 
question 



List the immediate results in your question List specific measures of the immediate 

results 



, Rewrite >0"r question with the immediate and longer term outcome variables operationali/ed. 
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3. List the background data you will need to collect and >our reason for wanting that information. 



Backgiound Info. Item Reason for Collecting ii 



4. Answer ih^si questions in the process of designing instruments to measure implementation. 



1. _ 

9 


Implementation Questions lo Answer 




3. 




4. 


How Will I Use This Information? 


1. 




9 




3. 




4. 


Who Should Be Involved in Instrument Design? 

First Draft of My Form 
(Complete on a Separate Sheetl 
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15. Fill out this data collection plan for one piece of effort or process information and one outcome 
(effectiveness) data collection instrumeru you will use to complete your evaluation. 



Data Collection Plan 


Instrument 


What it 
Measures 


Frequency of 
Collection 


Who Collects 


Accuracy of 
Data 


\_ 










2. 










3. 











16. Describe your plan for pilot testing your measurement instruments and ensuring that your data 
collection system is workable. 

INSTRUMENT PILOT TEST PLAN 



1. What are the instruments I will be using? 

2. Who will read them to niake sure they are appropriate for my audience? 

3. Where will 1 find a sample of 20-30 people to fill out the instruments'^ 

4. How will I know if the instruments are producing reliable information? 

5. How will I know if the instruments are producing valid information? 
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17. 



Use the following foTii to summarize your data analysis plans: 



**'^THE QUESTION**'^ 



I. Who is interested in my results? 



2. What d'^scriptive information will I report? 

a. 

b. 

c. 

d. 

e. 

3. What methods for analyzing relationships will I use? 

a. Contingency tables? 

b. Comparisons of Means? 

c. Correlations? 



4. ''ow will I define a differer.ce that is meaningful in a practical sense? 



5. Which test of significance will I use? 



Why? 
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18. Use this outline to help you in preparing to report your results. 

REPORTING RESULTS 



1. Write doun your evaluation question here: 

2. Draw one graph or table to display the results you hope you will find. Be optimistic here. 
It's your chance to have the data come out just as you want them. 

3. Using the visual displays you have created, plan a presentation for your audience. In the 
presentation, tell them: 

a. What you set out to learn— your question. 

b. What you actually found. 

c. What ycu think these results mean to this audience. (What you think should 
be done in light of this information.) 

Presentation Outline 
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Appendix B 

Survey Instrument 




Survey Instrument 



This Appendix contains one version of the self-report survey instrument used each year in the 
NIDA "Monitoring the Future" study to collect information from a sample of over 17,000 high 
school seniors in the United States. Four additional questions related to drug and alcohol use were 
selected from other versions of the survey instrument and are included after the survey form. Results 
of the survey are published annually by the National Institute on Drug Abuse (NIDA) and are 
available from ADAMHA Office for Substance Abuse (OSAP) Clearinghouse. You can select items or 
sets of items from this survey to use in your evaluation study if these items are appropriate to your 
evaluation questions. The instrument can also be consulted for format and style if >ou are creating 
your own questionnaires. By using items from this survey to measure drug-related and other out- 
comes, you can compare the results from people in your prevention evaluation project with the na- 
tional trends and rates reported in this study. Items relating to drug and alcohol use begin in part B 
on page 5 of the survey. 

At this writing, the most recent published report of this study available from NIDA is: 

Drug Use among American High School Students, College Students, and Other Young Adults, Na- 
tional trends through 1985. Lloyd D. Johnston, Patrick M. O'Malley, Jerald G. Bachman. Rockville, 
Maryland: National Institute on Drug / buse, 1986. 
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(jiji]@oi)Dft@irSinii) the future 

a continuing stuciy of the lifestyles and values of youth 



This questionnaire is part of a nationwide study of high school seniors, conducted 
each vear by the University of Michigan's Ins atute for Social Research. The ques- 
tions ask your opinions about a number of things-the way things are now and the 
way you think they ought to be in the future. In a sense, many of your answers 
on this questionnaire will counc as "votes" on a wide range of important issues. 

If this study is to be helpful, it is important that you answer each question as 
thoughtfully and frankly as possible. All your answers will be kept strictly confi- 
dential, and will never be seen by anyone who knows you. 

This study is completely voluntary. If there is any question that you or your 
parents would find objectionable for any reason, just leave i* blank. 

In a few months, we would like to mail each of you a summary of the nationwide 
results from this study. Also, in about a year we would like +0 mail another ques- 
tionnaire to some of you, asking about how your plans have worked out and what's 
happening in your lives. 

In order to include you in these mailings, we ask for your name and address on a 
special form at the end of this questionnaire. This form is to be torn out and handed 
in separately. Once the address form and the questionnaire have been separated, 
there is no way they can be matched again, except by using a special computer tape 
at the University of Michigan. The only purpose for that tape is to match a follow- 
up questionnaire with this one. 

Other seniors have said that these questionnaires are very interesting and that they 
enjoy filling them out. We hope you will too. Be sure to i-ead the instructions on 
the other side of this cover page before you begin to answer. Thank you very much 
for being an import-xnt part of this project. 

1986 - 1987 

INSTITUTE FOR SOCIAL RESEARCH 
THE UNIVERSITY OF MICHIGAN 
ANN ARBOR, MICHIGAN 
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INSTRUCTIONS 



1. This is not a test, so there are no right or wrong answers; we would 
like you to work fairly quickly, so that you can finish. 

2. All of the questions should be answered by marking one of the answer spaces. 
If you don't always find an answer that fits exactly, use the one that comes 
closest. If any question does not apply w you, or you are not sure of what it 
means, just leave it blank. 

3. Your answers will be read automatically by a machine called an optical mark 
reader. Please follow these instructions carefully: 



• Use only the black lead pencil you have been given. 

• Make heavy black marks inside the circles. 

• Erase cleanly any answer you wish to change. 
•Make no other markings or comments on the 

answer pages, since they interfere with the 
automatic reading. (If you want to add a 
comment about any question, please use the 
space provided below.) 



These kinds of markings 
will work: # ^ 0 

These kinds of markings 
will NOT work: ® || 



0 



(THIS SPACE FOR WRITTEN COMMENTS) 
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10. 




1. Taking all things together, how would you say things 
are these days— would you say you're very hiq)py, 
pretty happy* or not too happy these days? 

O Very happy 
O Pretty happy 
O Not too happy 



2. The next questions ask about the kinds 

of things you might do. How often ^ f 

do you do each of the following? ^ i § ^ 

(Mark one circle for each line) & S S 

a. Watch TV QO0 

b. Go to movies 0O| 

c. Go to rock concerts 

d. Ride around in a car 
(or motorcycle) just for fian 

e Participate in community affeirs or 
volunteer work 

f. Play a musical instrunrHriitor sing 

g. Do creative writing 

h. Actively participate in sports, 
athletics or exercising 

i. Do art or craft work 

@ P F 

j. Work around the house, yard, garden, P S ^ 
car tc pOSOO 

f& i i 

k. (j€t together with friends, informally 

1. (roshoppingor window-shopping . 

i 

m. Spend at least an hour of M p |« 

leisure time alone. ^ _ _ 

i S 1 

n. Read books, magazines, or newspapers 

z ^ u 

0. (Jototavems, bars or nightclubs SolSof 

p. (jo to partiesor other social affairs BOfiOC 




3. How much do you agree or disagree with each of the 
foUowing statements? (Mark one circle for each line.) 




a. In the United States, we put too much f 
emphasis on making profits and not cj ^ ^ ^ 
enough on human well-being f||®ff®|^ 

b. People are too much concerned with 
material things these days 

i 

c. Since it helps the economy to grow, 
people should be encouraged to buy ^ 
more 

i I i 

d. There is nothing wrong with M ^ 
advertising that gets people to buy 
things theydon't really need 5?^,^ 

W Si ,^ 

e There will probably be more shortages ^ || ^ 
in the future, so Americans will have P ^ S 
to learn how to be happy with fewer ^ | 
things" WDJI®! 



4. Bek>w are several ways that people have used to protect 
about serious social issues. How much do you approve or 
disapprove of these actions? (Mark one circle for each Lne.) 



a. Signing petitk)ns . 



b. Boycotting certain products or 
stores 



c. Lawful demonstrations 

d. Occupying buildings or factories . 

e Wildcat strikes 

f Blocking traffic- 

g. Damaging things |(DKD 

h. Personal violence 
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5. Do you think that you would prefer having: a mate for 
most of your life, or would you prefer not having a mate? 

© Definitely prefer to have a mate 
© Probably prefer to have a mate 
© Not sure 

© Probably prefer not to have a mate 
© Definitely prefer not to have a mate 

6. Which do you think you are most likely to choose in the 
long run? 

© Getting married 

© I have no idea 

© Not getting married 

® Am already married 

7. If you did get married (or are married). . . 

a. How likely do you think it is that you would stay 
married to the same person for life? 

® Very likely 
® Fairly likely 
© Uncertain 
© Fairly unlikely 
© Very unlikely 

b. How likely is it that you would want to have children? 

© Very likely 
® Fairly likely 
© Uncertain 
© Fairly unlikely 
© Very unlikely 

© AJready havechild(ren) 
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8. Imagine you are married and have 

no children. ^ ^ 

How would you feel about each of the foltow- x ^ 
ingworiting arrangements? (Mark one circle ^M^-^ 
for each line.) ^ f f 

a. Husband works full-time, wife doesn't work OXD®® 

b. Husband works full-time, wife wori<s about 
half-time ©®®® 

c. Both work (ull-time QXD(|)® 

d. Both work abouthalf-time ©®3)® 

e. Husband works about half-time, wife works 
full-time 0®®® 



f. Husband doesn't work, wife works full-time . ©©CD® 

9. Imagine you are married and have one 
or more pre-school children. 

How would you feel about each of the folk>w- 
ing working arrangements? (Mark one circle 
for each line.) 

a. Husbapd works full-time, wife doesn't work. . ®®<D® 

b. Husband works full-time, wife works about 



half-time (D®®® 

c. Both work full-time 0®(D® 

d. Both work abouthalf-time 0©®® 

e. Husband worths about hah'-time, wife works 
full-time 0@®® 



f. Husband doesnt work, wife works full-time . ®@®® 

10. Imagine you are married and have one or 
more pre-school children. Imagine also that 
the husband is working full-time and the 
wife does not have a job outside the home. 

How would you feel about each of these 
arrangements for the day to-day care of 



the child(ren)? 

a. Wifedoesallchiidcare ®@®® 

b. Wife does most of it ®@|^® 

c. Both do it equally ®@j)® 

d. Husband does mostof it ®@j|5® 

: i 

e. Husband does all of it 8l@(B© 
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11. Some people think about what's going on in government 
very often, and others are not that interested. How much 
of an interest do ^ou take in government and current events? 

® No interest at all 

© Very little interest 

® Some interest 

® A lot of interest 

® A very great interest 

12. Do you thinksome of the people runningthe government 
are crooked or dishonest? 

® Most of them are crooked or dishonest 

© Quite a few are 

® Some are 

® Hardly any are 

® None at all are crooked or dishonest 

13. Do you think the government wastes much of the money 
we pay in taxes? 

® Nearly all tax mono^ is wasted 
© A lot of tax money is wasted 
® Some tax money is wasted 
® A little tax money is wasted 
® No tax money is wasted 

14. How much of the time do you think you can trust the 
government in Washington to do what is right? 

® Almost always 
© Often 
® Sometimes 
® Seldom 
® Never 

15. Do you feel that the people running the government are 
smart people who usually know what they are doing? 

® ITiey almost always know what th^ are doing 
© They usually know what th^ are doing 
® They sometimes know what they are doing 
® They seldom know- what th^ are doing 
® They never know what th^ are doing 

16. WoukJ you say the government is pretty much run for a few 
big interests kwking out for themselves, or is it run for 
the benefit of all the people? 

® Nearly always run for a few big interests 
© Usually run for a few big interests 
® Run some for the big interests, some for the people 
® Usually run for ths benefit of all the people 
® Nearly always run for the benefit of all the people 



17. Have you ever done, or do you plan to do, ^ 

the fotk^wing things? ^ a ^ 

(Mark le circle for each line) S ? 

Q 

a Voteinapublicelection? 

b. Write to public officials? 

c. Give money to a political candidate or ^ ^ ^ 

cause? ii®lR £ 

P ^ 1^ 

d. Work in a political campaign? W 

B W U 

e. Participate in a lawful demonstration? . . . m®W 9 

I i i 

f. Boycottcertainproductsorstores? 9®t) 8) 




18. How much do you agree or disagree with each 
of the following statements? (Mark one 
circle for each line.) 



a The U.S. should begin a gradual ^f.^ f 

program of disarming whether other q ^ ^ ^ - 
countriesdoornot II^H^^^ 

b. There be times when the U.S. ^ ^ ^ 
should go to war to protect the ^ N ^ 

rights of other countries Qki)||®B 

III 

c The U.S. should be willing to go to ^ ^ ^ 
war to protect its own economic ^ ^ 

interests P®'^^ 

d. The only good reason for the U.S. to P ^ 
go to war is to defend against an ^ % 0 

attackonourowncountry O®Q|0dD 

1 i 

&5 yj'.^ 

e The U.S. does not need to have greater m p V ' 
military power than the Soviet Union K^p^^^ 

f. The U.S. o'jght to have much more Si i| |v 
military power than any other 8 || 
nation in th'^ world Sj®fl&©4J 

g. Our present foreign policy is based on ^ ^ 
our own narrow economic and power ^ |^ 
interests 

h. Servicemen should obey orders ^ ^ § 
without question fii®4^®iEf 
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19. Tlib section detb with acth^whfch may be agau^ 
rtiles or against the law. We hope you will answer all of 
these questions. However, if you find a question which 
you cannot answer honeetty. we would prefer that you 
leave it blenJc Remember, your answers will never be 
connected with your name. 



During the LAST 12 MONTHS, 
how often have you. . . 



a. Argujd or had a fight with either of 
your parents 



b. Hit an instructor or supervisor . 



c Gotten into a serious fight in school 
or at work 



d. Taken part in a fight where a group 
of your friends were against another 
group 



e Hurt someone badly enoug^i to need 
bandages or a doctor 



f. Used a knife or gun or some other 
thing (like a club) to get something 
fromaperson 



g. Taken something not belonging to 
you worth under$50 



h. Taken something not belonging to 
you worth over $50 Bq 

i. Taken something from a store without 5 
paying for it Hq 

j. Taken a car that didn't belong to 
someone in your family without 
permission of theowner BoBol 

k. Taken part of a car without 

permissionoftheowner 



1. Gone into some house or building 
when you weren't supposed to b6 
there 



m. Set fire to someone's property on 
purpose 



n. Damaged school property on purpose. 

0. Damaged propeiiy at work on 
purpose 



p. Gotten into trouble with police 
because ofsomething you did . . . 
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20. The next questfons are about some things 
which may have happened TO YOU. 

During the LAST 12 MONTHS, ^ | 

how often. ... ^ ^ S 

a. Has something of yours (worth under o ^ 
$50) been stolen? 

b. Has something of yours (worth over 
$50) been stolen? _ 

c. Has someone deliberately damaged 
your property (your car, 

clothing,etc.)? SoSoS 

d. Has someone iryured you with a P ^ ^ 
weapon (like a knife, gun, or club)? £o§)oS 

S W M 

e. Has someone threatened you with a § ^ 
weapon, but notactuaIlyii\jured you? . . . BO^OQ 

f. Has someone iryured you on purpose ^ § ^ 
withoutusingaweapon? !5§C)0O© 

M ^ P. 

F Has an unarmed person threatened p P 
you with iryury, but not actually ^ fi 

injured you? bo^OO 



21. "How difficult do you think it would be f 

for you to get each of the following §^ f 

types of d^-ugs, if you wanted some? Sg^ti^ 

(Mark one circle for each line.) J 5^ ^ 

a Maruuana(pot grass) ||®|||®|^ 

b LSD l(2)l©i 

ri 

c Some other psychedelic (mescaline, 

peyote,psilo(ybin,PCP,etc) BCzJOKOI 

d. Amphetamines (uppers, pep pills. 

bennies, speed) S®|©l 

e Barbiturates (downers, goofballs, 

reds, yePows, etc) B©B©I 

i Tranquilizers ■©B0I 

g. Cocaine ■®B0| 

h. Heroin (smack, horse) mctm^ 

i. Some other narcotic (methadone, 
opium, codeine, paregoric, etc.) m(^Wk(^ 
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PARTB 



1. Have you ever smoked cigrarettes? 

©Never-GO TO QUESTION 8 

® Once or twice 

® Occasionally but not regularly 
© Regularly in the past 
© Regularly now 

2 How frequently have you smoked cigarettes during the 
past 30 days? 

© Not at all 

® Less than one cigarette per day 
© One to five cigarettes per day 
© About one-half pack per day 
(5) About one pack per day 
© About one and one-half packs per day 
® Tv/o packs or more per day 

8. Next we want to ask you about drinking alcoholic beverages, 
including beer, wine, and liquor. 

Have you ever had any beer, wine, or liquor to drink? 

© No-GO TO THE TW W THE NEXT OOUJIIN 

©Yes 

4. On how many occask>ns have you had ^ ^ | $ s 
alcoholic beverages to drink. . . ii^ §<i<5^ 
(Mark one circle for each lina) ^ ^ 

a ...inyour lifetime? OOOOOOO 

b. ...during the last 12 months?... CX>C)0600 

c. ...duringthelastSOdays? OOOOOOO 

5. On the occaswns that you drink alcohoUc beverages, how 
often do you drink enough to feel pretty high? 

© On none of the occasions 
® On few of the occasions 
© On about half of the occasions 
© On most of the occasions 
® On nearly all of the occasions 

6. Think back over the LAST TWO WEEKS. How many times 
have you had five or more drinks in a row? (A "drink" is 
a giasB of wine, a bottle of beer, a shot glass of liquor, or a 
mixed drink.) 



® None 
© Once 
©T'Arice 



© Three to five times 
© Six to nine times 
© Ten or more times 
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I no 



7. On how many occask>ns (if any) ^ ^ 
have you used marijuana (grass, § ^ § i i 
pot) or hashish (hash, hash oU). . . ijJJ //^ 
(Mark one circle for each lina) ^ 

a ...in your lifetime? — OOQOOOO 

b. ...duringthe last 12 months? ... OOOOOOO 

c. ...duringthelastSOdays? OOOOOOO 

8. On how many occask>ns (if any) 

have you used LSD ("acki"). . . ^ a. ^ ^ ^ 

a ...in ycur lifetime? Q'OOOOOO 

b. ...during the last 12 months?... OOOOOOO 

c. ...duringthelastSOdays? OOOOOOO 

9. On how many occask)ns (if any) 
have you used pe^chedelk^s other 
than LSD (like mescalirte, peyote, 
p8ik)cybin.PCP)... cs,<oc,i?§- 

a ...in your lifetime? OOOOOOO 

b. ...during the last 12 months?... 6O(^0OO 

c. ...duringthelastSOdays? ...... OOOOOOO 

10. On hew many occasions (if any) 
have you used cocaine (sometimes 

called "coke", "crack", "rock"). . . 5? § ^ 

a ...m your lifetime? OOQO0O0 

^ % ^' 

b. ...during the last 12 months? . . . 0O©O©OO 

M h ,^ ± 

c. ...during the lastSOdays? 0O0O0OO 

11. Amphetamines have been prescribed by doctors to help 
people lose weight or to give people more energy. They 
are sometimes c?lled uppers, ups, speed, bennies, dexies, 
pep pills, and diet pills. Drugstores are not supposed 

(1986^7 Base Year. Forms 2-5 • Part B) 
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to sell them without a prescription from a doctor. 
Amphetamines do NOT include any non-prescription drugs, 
such as over-the-counter diet pflls (like Dexatrim®) or 
stay-awake pills (like No-Do2«), or any mail-order drugs. 
On how many oecarions (if any) 
have you taken amphetamines on 
your own-that is, without a doctor 
Idling you to take them* . . 

a. ...in your lifetime? 

b. ...duringthe last 12 months? 

c. ...duringthe last 30 days? 

12. On how many occaskms (if any) liave you used quaaludes 
(Quads» soapers, methaquakme) ov* your own-that is, without 
a doctor teOinc: you to take thenL . . 

a. ...in your lifetime? ^OC^OQO^ 

b. ...duringthe last 12 months? 0ODO0OO 

)$ 1 C 

a ...euringthelast30days? filOOOOoS 

13. Barbiturates are sometimes prescribed by doctors to help 
people relax or get to sleep. Hiey are sometimes called 
downs, downers, goofbaUs, yeUows, reds, blues, rainbows. 
On how many occaskms (if any) have you taken barbiturates 
on your own-that is, without a doctor telling you to take 



them. . 



a. ...in your lifetime? -OOOOOOO 

b. ...duringthe last 12 months? -OO&OOOG 

c. ...duringthe Iast30days? ibo<DOOOO 

14. Tranquilizers are sometimes prescribed by doctors to calm 
people down, quiet thdr nerves, or relax their muscles. 
Librium, Valhmi, and Miltown are all tranquilizers. 

On how many occaskMis (if any) have you taken tranquilizers 
on your own-that is, without a doctor telling you to take 
them. . . ^ A 

a. ...inyourlifetime? QOOOOOO 

b. ...duringthe last 12 months? ^OOOQOO 

c. ...duringthe Iast30 days? OOOOOOO 

15. On how many occasions (if any) have you used heroin 
(smack, hone, skag). . . 

a. ...inyourlifetime? OOOOOOO 

b. ...duringthe last 12 months? (DOOOOOO 

c. ...duringthe last30 days? OOOOOOO 
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16. There are a number of narcotks other than heroin, such as 
methadone, opium, morphine, codeine, demerol, paregoric, 
talwin, and laudanum. These are sometimes prescribed by 
doctors. 

On how many occasions (if any) have you taken narcotics 
other than heroin on your own-that is, without a doctor 
telling you to take them. . . 

a. ...inyourlifetime? OOOOOOO 

I i I 

b. ...duringthe last 12 months? ^^^^^^^ 

c. ...duringthe Iast30 days? ^oJlowO^ 

17. On how many occasions (if any) have you sniffed glue, or 
breathed the contents of aerosol ^ray cans, or inhaled any 
other gases or ^niys in order to get high. . . 

a. ...inyourlifetime? (9O0OpOO 

1- i i 

b. ...duringthe last 12 months? jSOOOOOO 

c. ...duringthe last30 days? ©OOOOOO 
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1. In what year were you bom? 

©Before *66 ©1967 ©1969 
©1966 ©1968 ©1970 

2. In what month were you bom? 



©1971 

© After 1971 



© January © April © July © October 

© February ® May ® August (Q) November 
© March ® June ® September @ December 

3. What is your sex? ® Male ® Female 

4. How do you describe yourself? 

© American Indian 

© Black or Afro-American 

© Mexican American or Chicano 

© Puerto Rican or other La* in American 

© Oriental or Asian American 

© White or Caucasian 

© Other 
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5. Where did you grow up mostiy? 
© On a farm 

© In the country, not on a farm 

© In a small city or town (under 50,000 people) 

© In a medium-sized city (50,000 - 100»000) 

® In a suburb of a medium-sized city 

© In a large city (100,000 • 500,000) 

© In a suburb of a large city 

© In a very large city (over 500.000) 

© In a subur b of a very large city 

© CanV S2^y; mixed 

6. What is your present marital statu«? 



® Married 
©Engaged 



Q Separated/divorced 
® Single 



7. How many brothers and sisters do you have? (Include 
step-brothers and sisters and half-brothers and sisters.) 



a. Older brothers and sisters . . . 

b. Younger brothers and sisters 



: SUSHI 



7c. Which of the following people live in the same 
household with you? (Mark all that apply.) 



O I live alone 

O Father (or male guardian) 
O Mother (or female guardian) 
O Brother(s) and/or sister(s) 
O Grandparent(s) 



O My husband/wife 
O My children 
O Other relative(s) 
O Non-relative(s) 




8. What is the highest level of schooling your father completed? 

O (Completed grade school or less 
O Some high school 
O Completed high school 
O Some college 
O (Completed college 

O Graduate or professional school after college 
O Don't know, or does not apply 

9. What is the highest level of schooling your mother 
completed? 

O Completed grade school or less 
O Some high school 
O (Completed high school 
O Some college 
O (Completed college 

O Graduate or professional school after college 
O Dont know, or does not apply 
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10. Did your mother have a paid job (half-time or more) 
during the time you were growing up? 

©No 

© Yes, some of the time when I was growing up 

0 Yes, most of the time 

© Yes, all or nearly all of the time 

11. How would you describe your political preference? 

(Mark one.) 

© Strongly Republican 

® Mildly Republican 

© Mildly Democrat 

© Strongly Democrat 

® American Independent Party 

© No preference, independent 

©Other 

© Don't know, haven't decided 

12. How would you describe your political beliefs? (Mark one) 

© Very conservative 
® (]!onservative 
® Moderate 
©Liberal 
® Very liberal 
® Radical 

® None of the above, or don't know 

13. The next three questions are about religion, 
a. What is your religious preference? 



© Baptist 

® Clhurches of Christ 

® Disciples of Chris^ 

© Episcopal 

® Lutheran 

® Methodist 

© Presbyterian 

® United Church of Christ 



© Other Protestant 
© Unitarian 
® Roman Catholic 
® Eastern Orthodox 
® Jewish 

® Latter Day Saints 
® Other religion 
® None 



b. How often do you attend religious services? 

© Never 
® Rarely 

® Once or twice a nrwnth 
® About once a week or more 



How important is religion in your life? 

© Not important 
® A little important 
® Pretty important 
© Very important 
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14. When are you ,nosi likely to graduate from high school? 

® By this June 
® July to January 
® After next January 

® Don't expect to graduate 

15. Wh> the following best describes your present high 
school program? 

® Academic or college prep 
® General 

® Vocational, technical, or commercial 
® Other, or don't know 



16. Compared with others your age 
throughout the country » how do 
you rate yourself on school ability? 

17. How intelligent do you think you 
are compared with others your 



age? 




18. During the LAST POUR WEEKS, 
how many whole d^ys of school 
have you missed. . . 



i 



Q ^ ^ ® 
cvj ^ <e ^ 




a. Because of illness 

b. Becauseyou skipped or "cut' 

c. For other reasons 



19. During the last four weeks, how often have you gone to 
school, but skipped a class when you weren't supposed to? 

® Not at all 

® 1 or 2 times 

® 3-5 times 

® 6-10 times 

© 11-20 times 

® More than 20 times 

20. Which of the foUowing best describes your average grade 
so far in high school? 

® A (93-100) 
® A- (90-92) 
® B+ (87-89) 
®B (83-86) 
® B- (80-82) 
®C+ (77-79) 
®C (73-76) 
®C-(70--72) 
® D (69 or below) 
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21. How likefy b it that you will do each 
of the foUowing things after high 
school? (Mark one for each line.) ^^^^^^ 

#### 

a Attend a technical or vocational q ^ q 
school Q)®9}® 

b. Serve in the armed forces O®®® 

c. Graduate from a two-year college 

program ®(D®® 

d. Graduate from college (four -year 

program) ®©®© 

e. Attend graduate or professional 

school after college ®©®® 



22. Suppose you could do just what you'd like and 
nothing stood in your way. How many of the 
following things would you WANT to do? 
(Mark ALL that apply.) 

O a. Attend a technical or vocational school 
O b. Serve in the armed forces 
O c. Graduate from a two-year college program 
O d. Graduate from college (four-year program) 
O e. Attend graduate or professional school 

after college 
O f- None of the above 

23. On the average over the school year, how many hours per 
week do you work in a paid or unpaid job? 

® None 

© 5 or less hours 
® 6 to 10 hours 
® 11 to 15 hours 
® 16 to 20 hours 
® 21 to 25 hours 
® 26 to 30 hours 
® More than 30 hours 



24. During an average week, how much 
money do you get from. . . 



a. A job or other work 



■OOOOOOOO^ 



b. Othersources(allowances,etc.) OOOOOOOO^ 

25. During a typical week, on how many evenings do you go 
out for fun and recreation? 

O Less than one 

O One 

O IVo 

O Three 

O Four or five 

O Six or seven 
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26. On the average, how often do you go out with a date (or 
yovT spouse, if you are married)? 



® Never 

© Once a month or less 
0 2 or 3 times a month 



® Once a week 

© 2 or 3 times a week 

® Over 3 limes a week 



27. Durin ; an average week, how much do you usuaDy drive a 
car, truck, or motorcycle? 



® Not at all 
© 1 to 10 miles 
© 11 to 50 miles 



® 51 to 100 miles 
© 100 to 200 miles 
® More than 200 miles 




28. Within the LAST 12 MONTHS how many times, if any. 
have you received a ticket (OR been stopped and warned) 
for moving violations, such as speeding, running a stop 
light, or improper passing? 

® None-GO TO QUESTION 80 

® Once 
® Twice 
® Three timer 
® Four or mc.e times 

29. How many of these tickets or warnings 
occurred after you were. . . a a. o ^ ^ 

a. Drinkingalcoholic beverages? 0>®S)®© 

b. Smokingmaryuanaorhashish? (i®®®dCi 

c Using other illegal drugs? <J®(^^ 



M We are interested in any acckients which occurred while 
you were driving a car, truck, or motorcycle. ("Acckients" 
means a coUiskm invdving property clomage or personal 
injury-not bumps or scratches in parking k>t8.) 

During the LAST 12 MONTHS, ho. .mny accWents have 
you had while you were driving (whether or not you were 
responsible)? 

® None-GO TO QUESTION » 
® One 
© Two 
© Three 
® Four or more 



31. How many of these accidents 
occurred after you were. . . 

a. Drinking alcoholic beverages* 

b. Smokingmarijuanaor hashish? 

c. Usingothcr illegal drugs? 




What is, or will be, your branch of service? 

® Army © Marine Corps ® Cbast Guard 

© Navy ® Air Force ® Uncertain 

33. Do you expect to be ar. officer? 

© No ® Uncertain © Yes 

34. Do you expect to have a career in the Armed Forces? 
® No © Uncertain ® Yes 

PAirrD 

This section nks for your views and feelings about a 
number of dii^erent things. 



1. How satisfied are you with your life as a whole 
these days? 

® Completely dissatisfied 

© Quite dissatisfied 

® Somewhat dissatisfied 

® Neither, or mixed feelings 

® Somewhat satisfied 

® Quite satisfied 

© (Completely satisfied 

I llKse next questwns ask you to guess how well yoii might 
do in several different situations. 
(Mark one circle for each line.) 



How good do you think you would 
be... 



a. Asahusbandorwife? 

b. As a parent? 



•. i i i 

1 11 i 

c. As a worker on a job? !®®3l)®!D 9| 



3. Some people think a k)t about the social problems of the 
natk>n and the worW, and about how they might be solved. 
Others spend little time thinking aba'Jt these issues. How 
much do you think ah^ut such things? 

® Never 
© Seldom 
© Sometimes 
® Quite often 
® A great deal 
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4. Now we'd like you to make some ratings of how honest 
and mora] the people are who run the following 
organizations (Mark one circle for each line.) 
To what extent are there problems ^ 
of dishonesty and immorality in r ^ ^ 

the leadership of. . . J^/li? / 

a Large corporations? 



b. M^r labor unions? 



c. The nation's collies and 
universities? 



d. The nation's public schools? 



e. Churches and religious organi- 
zations? 



f. The national news media (TV, 
magazines, newsservices)? 



g. The Presidency and tlie admin- 
istration? 



h. Congress-that is, the U.S. 
Senate and House of 
Representatives? 



i. The U.S. Supreme Court? 



j. All the courts and the justice 
system in general? 



The police and other law 
enforcement agencies? . . . 



1. The U.S. military? 



5. How much do you agree or disagree with each of the 
following statements? (Mark one circle for each lina) 



a There is too much competition in 
thissociety 



b. Too many young people are sloppy 
about their grooming and clothing 
and justdon'tcarehowthey look . . . . 



a There is too much hard rock music 
on the radio these days 



d. ;*«ople should do their own thing, 
even ifother people think it's strange 

e. I get a real kick out of doing things 
that are a little dangerous M&fM^ 
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f. I like to test myself every now and §f^^Sf 
then by doing something a little cHH^ 

nsi<y ©©®o© 

g. 1 take a positive attitude toward myself . . .C?)©® Q® 

h. I feel I am a person of worth, on an 

equal plane with others 0©®©® 

f; ' 

i. I am able lo do things as well as most to- 
other people p©®o® 

j. On the whole, Tm satisfied with myself .... (D©@©® 

k. I feel I do not have much to be proud of ... . Q)©® ©® 

1. Sometimes I think that I am no good at pJl .Q©®(i)® 

nx i feel that I can't do anything right (jD®®©® 

n. I feel that my life is not very useful ®(D@0® 

6. How many of your friends wouW you 
estimate. . . 

a Smoke cigarettes? ®©®©® 

b. Smoke marijuana (pot, grass) or hashish? ®@®©® 

c. Take LSD? ®©®©® 

d. Take other psychedelics (mescaline, 
peyote,PCP,etc.)? ®©®©® 

a Take amphetamines (uppers, pep pills, 

bennies, speed)? (D©®©® 

f. Take quaaludes (quads, methaqualone)? 0©®©® 

g. Take barbiturates (downers, 

goofballs, reds, yellows, etc.)? ®©®©® 

h. Take tranquilizers? ©@@®® 

i. Take cocaine? ®©®®® 

J. Take heroin (smack, horse)? ®©®®® 

k. Take other narcotics (methadone, 

opium, codeine, paregoric, etc.)? (D©®®® 

1. Use inhalants (sniffing glue, aerosols, 

laughing gas, etc.)? ®(i>®(D® 

m. Drink alcoholic beverages (liquor, beer, wine)? .®©@®® 
n. Get drunk at least once a week? ^©©(5® 
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Lately ther« has been some attention paid to certain 
drugs, including "crack** (cocaine in chunks or rocks). 



1. How many of your friends 
would you estimate. . . 

a. Take "crack" cocaine? . , . 



^ -t c9 ^ '•C 



b. Take TCP (angel dust, ci-ystal, peace pill, 
killer weed, su,.3rgrass, crystal cyclone)? . 

c. Ta^e amyl or butyl nitrites (poppers, 
snappers, Locker Room, Vaporole, Rush, 
Kick, Bullet)? 

2. How much do you think people risk 
harming themselves (physically or 
in other ways) if they., . . 




a. Try cocaine in powder form 
once or twice 



^ ■£ i§ T :^ 



b. Take cocaine powder occasionally . . 

c. Ta^e cocaine powder regularly .... B? 

d. Try "crack" cocaine once or twice. . 

e. Take "crack" cocaine occasionally . . 

f. Take "crack" cocaine regularly .... ■k| 

g. Try PCP once or twice K| 

3. On how many occasions (if any) 
have you used "crack" cocaine? 

a. . . .in your lifetime? 

b. . . .during the last 12 months? . . . 

c. . . .:iuring the last 30 days? 




4. On how many occasions (if any) 
have you used PCP? 

a. . . .in your lifetime? 

b. . . .during the last 12 months? . . 

c. . . .during the last 30 days? .... 

5. On how many occasions (if any) have 
you used amyl or butyl nitrites? ^ *p oi 




a. . . .in your lifetime? 

b. . . .during the last 12 months? . . . 

c. . . .durii '7 the last 30 days? 



II 
I II 



6. When (if ever) did you FIRST do 
each of the following things? 



a. Try "crack" cocaine O 

b. Try any other form of cocaine . O 

c. Try PCP O 

d. Try amyl or butyl nitrites . . . O 



How difficult do you think it would 
be for/ you to get each of the 
following, if you wanted some? 

a. "Crack" cocaine 

b. Cocaine in powder form 

c. PCP 

d. Amyl or butyl nitrites 



to o> ^c-sy 




8. 



9. 



What is your best guess about whether your parents 
think you drink alcoholic beverages (beer, wine, liquor)? 

© They feel sure I don't drink 

® They think I probaby don't drink 

® I don't know, or they differ in whether they think I do 

® They think I probably drink 

® They feel sure (or know) that I drink 

What's your best guess about whether your parents 
think you smoke marijuana? 

® They feel sure I don*t 

© They think I probaby don't 

© I don't know, or they differ in whether they think I do 

© They think I probably do 

© They feel sure (or know) that I do 



10. Do you favor or oppose a military draft at the present 
time? 



© Strongly favor 

© Mostly favor 

© No opinion, or mixed 



@ Mostly oppose 
© Strongly oppose 



11. 



Do you think any military draft in the U.S. should 
include women as well as men? 



O Yes 



O Uncertain © No 



-11- 
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12. Some people like school very much. Others don't. 
How do you feel about going to school? 

© I like school very much © I don't like school very much 
© I like school quite a lot © 1 don't like school at all 
© I like school some 

(1387 Bam V»tr Form 2 • Pari £) 
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13. About how many hours do you spend in an average 
week on all of your homework including both in 
school and out of school? 



O 0 hours 
O 1-4 hours 
O 5-9 hours 



O 10-14 hours 
O 15-19 hours 
O 20-24 hours 



O 25 or n^ore 
hours 



14. To what extent have you 

participated in the following school 
activities during this school year? 

a. . . school newspaper or yearbook 



.music or other performing arts 



.athletic teams 



d. . . .other school clubs or activities 



15. In general, how much say or influence 
do you feel each of the following has 
on HOW YOUR SCHOOL IS RUN? 
(Mark one circle for each line.) 




a. The principal. 



b. The teachers 



c. The students .... 

d. Parents of students 




16. Have you had any drug education courses or 
lectures in school? 

© No-GO TO QUESTION §1 

® No, and I wish I had-GO TO QUESTION^ 

(D Yes 

17. Would you say th&t the information about dn:*f8 that 
you ^ecoived in school classes or progranu has. . . 

© Made you less interested in trying drugs. 
@ Not changed your interest in trying drugs. 
® Made you more interested in trying drugs. 

18. How many of the following drug education 
experiences have you had in high school? (Mark all 
that apply.) 

O A special course about drugs 
O Films, lectures, or discussions in one of my 
regular courses 

O Films or lectures, outside of my regular courses 
O Special discussions ("rap" groups) si jut drugs 

19. Overall, how valuable were the experiences io you? 



© Little or no value 
@ Some value 



© Considerable value 
© Great value 



I Dui ing the LAST TWO WEEKS, how many 
times (if any) have you driven a car,, truck, ^ 

^ O ei <6 ^ 

a. drinking alcohol? . . . BOBOijO 



or moturcycie after. 



b. having 5 or more drinks in a row? . 




21. During the LAST TWO WEEKS, how many 
times (if any) have you been a passenger 
in a car. . . « 



< O fCr 0^ ^ 

a. when the driver had been drinking? . BOBOpO 

b. when you think the driver had 5 or p p |' 
more drinks? OOSODO 



22. How often do you. . . 

a. Eat breakfast? 

b. Eat at least some g^reen vegetables? 

c. Eat at least some fruit? 



d. Exercise vigorously O'og^ng, swimming, 
calisthenics, or any other active sports)? . 

e. Get at least seven hours of sleep? . . 

f. Get less sleep Uian you thmk you should? 




23. When you drive a car. how often ^:|^cf #^ / 
do you wear a seatbelt? 



24. When you are riding in the front 
passenger seat of a car. how p 
often do you wear a seatoelt? .... 5©({)®(6 (8) 





Thank you for taking the time to 
answer these questions. We hope 
you found them interesting. We are 
eager to tabulate your ans'vers 
along with those of other high 
school seniors throughout the 
nation. 
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WHY YOUR NAME AND ADDRESS? 

/ 

As we told you earlier, we'd like to send 
you a summary of the nationwide results 
of the present study, and in about a year 
we want to mail a similar questionnaire 
to some of you. In order to include you 
in these follow-ups, we would like to have 
an address where information will be sure 
to reach you during: the coming year. 



HOW IS CONFIDENTIALITY 
PROTECTED? 

• The information on this page will be 
used ONLY for mailing, and will always 
be kept separate from your answers. A 
special Grant of Confidentiality from 
the U.S. government protects all 
information gathered in this research 
project. 

• The questionnaire and address pages 
will be collected separately, sealed 
immediately in separate envelopes, and 
sent to two different cities foi 
processing. 

• Once a questionnaire and address page 
have been separated, there is no way 
they can be matched, except by using 
a special computer tape at the Univer- 
sity of Michigan. That tape contains 
the two DIFFERENT numbers that 
appear on the back of this address 
page and on the back of the question- 
naire. These numbers will be used 
ONLY to match a follow-up question- 
naire with this one. 



Before filling out this address page, please separate it from 
the rest of the questionnaire by FOLDING ALONG THE 
PERFORATED LINE AND TEARING CAREFULLY. 



Please PRINT your name and the address where you can most 
likely be reached during the coming: year. 

Mr. 

Miss 

Ms FIRST NAME INITIAL LAST NAME " 

Mrs 

STREET 

CITY 

STATE ZIP 

TELEPHONE NO ( ) - 

AREA 

In case we should have trouble getting mail to you, if you move, 
please PRINT the name and address of one other person (with a 
different address than your own) who will know where to reach 
you in the future. (Examples of such a person: aunt or uncle, older 
sister or brother, or close friend.) 

Mr 

Miss 

Ms FIRST NAME InITIAL " LAST NAMe' 

Mrs 

STREET 

CITY 

STATE ZIP 

TELEPHONE NO ( ) ~ 

AREA 



THANK YOU AGAIN FOR YOUR HELP 
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''Attitudes of Friends Toivard Drug Use" 

8. How do you think your CLOSE FRIENDS ^ 

feel <or would feel) about YOU doing each ^ | 

of the following things? {Mark one circle | ^ 

for each line.) * 1 ^ 

a. Smoking one or more pack of cigarettes 

per day (D®© 

b. Trying marijuana (pot grass) once or twice. . . 

c. Smoking marijuana occasionally H)©! 

I? 

d. Smoking njarijuana regularly 

e. Trying LSD once or twice ©©I^ 

t Trying an amphetamine (upper, pep pill, J 
bennie, speed) once or twice S®9 

_ . ^» 

g. Trying cocaine once or twice Q®^ 

i' i 

h. Taking cocaine occasionally fflCDB 

f ^. 

i. Taking one or twc drinks nearly every day . . . $©9 

j. Taking four or five drinks nearly every day . . 

k. Having ' ve or more drinks once or twice % ^ 
each weekend 

1 Driving a car after having 1-2 drinks. . . 

m. Driving a car after having 5 or more drinks . . t|)®i|) 




"Age of First Use of Drugs'' 

106. When (if ever) did you FIRST do each of the ^ ? 
following things? Don't count anything j ij^ 
you took because a doctor told you to. ^ oo ^^cf^cS^ 
(Mark one circle for each line.) ^ -^^S" 

i mm 

a Smoke cigarettes on a daily basis . O 

I I 

b. Try an alcoholic beverage— P ^ S 
more than just a few sips O -■■^-'^ 

c. Try marijuana or hashish O 

d. Tiy LSD Q 

e. Try a psychedelic other than LSD . O OpO^C 

f. Try amphetamines. O 

g. Try quaaludes Q 

h. Try barbiturates Q 

1 i ^ 

i. Try tranquilizers Q 

t. i i 

j. Try cocaine Q 

k. Try heroin Q Ofc 

1. Try any narcotic other than heroin. O 0| 

m. S^noke your first cigarette ... .0 Oj 

n. Drink enough to feel drunk 

or very high Q Ot)Ol 
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^^Disapproval of Drug Use" 

28. Individuals differ In whether or not they 
disapprove of people doing: certain things. 
Do YOU disapprove of people (who are ^ 
18 or older) doinfl: each of the following? ^ S 

(Mark one circle for each line.) / 

a. Smoking one or more packs of cigarettes < 
per day 

b. Trying maryuana (pot, grass) once or 
twice m(2A 

c Smoking marijuana occasion ally 

d. Smoking maryuana regularly 

e. Trying LSD once or twice 

f. Taking LSD regularly 

g. TVying heroin (smack, horse) once or twice 

h. Takingheroin occasionally 

i. Taking heroin regularly 

j. Trying a barbiturate (downer, goofbail, 
red,yellow, etc.) once or twice 

k. Taking barbiturates regularly P^^^ 

1. Trying an amphetamine (upper, pep pill, ^ ^ 
ben nie, speed) once or twice B(DB 

I I 

m. Taking amphetamines regularly 

n. Trying cocaineonceor twice |B©| 

0. Takingcocaine regularly 

p. Trying one or two drinks of an alcoholic g j 
beverage(beer, wine, liquor) %(t\ 

q. Takingone or two drinks nearly evervd^ . . . mc^ 

r. Taking four or five drinks nearly every 

day ilC2i 

s. Having five or more drinks once or twice 

each weekend "(©ili 
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"Perceived Risk of Drug Use" 

23. Toe next questions ask for your opinions on (he effects 
of usinff certain drugs and other substances. First, 
how much do you think people risk harming themselves 
(physically or in other ways), if they. . . 

a. Smoke one or more packs of ^ 5? c5 
cigarettes per day ® ®®® ® 

b. Try marijuana (pot grass) 

once or twice ® (D®® © 

c. Smoke marijuana occasionally . . .(p@®® ® 

d. Smoke marijuana regularly . . . .(pd)^® ® 

e. Try LSD once or twice ^^^^ ^ 

f. Take LSD re.?ularly Ikl)!© ® 

g. Try heroin (smack, horse) once 
or twice i MC^^MG) ^ 

h. Take heroin occasionally P^^^ ^ 

i. Take heroin regularly 0(D^® ® 

j. Try barbiturates (downers, * 
goorballs, reds, yellows, etc / 
once or twice (J)@®® ® 

k. Take barbiturates r^larly . . . .(Dd)®® ® 

1. Try amphetamines (uppers, pep & 

pills, bennies, speed) once or twice . J^ljS® ® 

m. Take amphetamines reg-jlarly . . .<i>(D^5® ® 

n. Try cocaine once or twice ®(D®® ® 

0. Take cocaine r^larly ®(D®® ® 

p. Try one or two drinks of an alco- 
holic beverage (beer, wine, liquor) .®(D®® ® 

q. Take one or two drinks nearly 

evciT day ®©®® ® 

r. Take four or five drinks nearly 

everyday (D(D®® ® 

s. Have five or more drinks once or 

twice each weekend (j(D®® ® 

t Take cocaine occasionally 0(D®® ® 

u. Use smokeless tobacco regularly 

(chewing tobacco, plug, dipping . .®®®® ® 
tobacco, snuff) 
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This table is reproduced with permission from tables of the Rand Corporation. Reprinted from Edwin 
B. Cox, ed., Basic Tables in Business and Economics. New York: McGraw-Hill, 1967. 
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Sample Sizes Required to Show Significance at 
the 95 Percent Level (One-Tailed Test) in 
Differences in Proportions in Groups of Equal Size* 



P2 pi = 05% 10% 15% 20% 25% 30% 35% 40% 45% 



10% 


418 


















20% 


84 


195 


791 














30% 


42 


67 


120 


269 


1064 










40% 


27 


37 


53 


83 


145 


319 


1237 






50% 


19 


24 


31 


42 


60 


92 


159 


344 


1311 


60% 


H 


17 


21 


27 


34 


45 


63 


95 


162 


70% 


li 


13 


15 


18 


22 


27 


35 


45 


62 


80% 


9 


10 


11 


13 


16 


18 


22 


27 


33 


90% 


7 


8 


9 


10 


11 


13 


15 


17 


20 



Note: pi = the proportion or percentage of a group having a particular characteristic. 

P2 = the comparable proportion observed in another group. (p2 should be assigned to the 
group having the higher proportion.) 



* Adapted from Table A-3, Joseph L. Fleiss, Statistical Methods for Rates and Proportions. New 
York: Wiley Interscience, 1973. 
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